Run each through the “vegetarian” options. I’m really curious how LIWC will look at the outputs of the models with relation to each other, and to the ground truth. Also get the counts of the occurrences of each prompt in the GT by star rating. My guess is that it won’t show up in some of the cases, which sets up the Twitter section really well.
4:15 Meeting
SBIRs
Fix duplicate entries in the DB topic file – done
Back up db – done
Create superclass that has most of the parts and then subclass the various implementations (Full, BuildView, ViewScript) – done
Work with Aaron on the stories/maps? Also, what is our plan for the paper? In process
WordPress has some serious lag. Need to back this up
GPT Agents
Delayed meeting until Friday. That should give me time to get the balanced data working and compare baseline models to the baseline data (American)
And the balanced data still isn’t working. I think that there are more paths to good reviews, so the GPT, even when fed balanced data generates unbalanced results. Training up single star models to verify this
Also, write a first pass on the introduction that uses the vegetarian Yelp as an example, and then set up to explain the method
Do I still need to train x-star models?
SBIRs
9:15 Standup
11:00 LAIC
Get the script running for the current map to show at the meeting today – done!
…clearly Biden was a net drag on McAuliffe. Overall, Virginians disapproved of Biden’s handling of the presidency by a 10-point margin, with nearly half saying they “strongly disapprove” — double the percentage who strongly approved. Nearly 3 in 10 Virginia voters said their vote was meant to express opposition to Biden, network exit polls found, compared to the 2 in 10 who said their vote was to express support for Biden. The economy was by far the most important issue driving Virginia voters, and people who put the economy at the top of their list favored Youngkin by a dozen percentage points. (Washington Post)
I just found this: https://github.com/google-research/tiny-differentiable-simulator It appears to be a NN-enhanced physics sim: “TDS can run thousands of simulations in parallel on a single RTX 2080 CUDA GPU at 50 frames per second:“ Here are the relevant papers:
“NeuralSim: Augmenting Differentiable Simulators with Neural Networks”, Eric Heiden, David Millard, Erwin Coumans, Yizhou Sheng, Gaurav S. Sukhatme. PDF on Arxiv
“Augmenting Differentiable Simulators with Neural Networks to Close the Sim2Real Gap”, RSS 2020 sim-to-real workshop, Eric Heiden, David Millard, Erwin Coumans, Gaurav Sukhatme. PDF on Arxiv and video
“Interactive Differentiable Simulation”, 2020, Eric Heiden, David Millard, Hejia Zhang, Gaurav S. Sukhatme. PDF on Arxiv
Finished training the balanced model and am re-running the original prompts
A really negative prompt will produce a low review distribution. Here’s an example of GPT generating reviews in response to a slightly negative set of prompts ([there are absolutely no vegetarian options], [there is not a single vegetarian option on the menu], [the menu has no vegetarian options]), compared with the ground truth of the Yelp database returning reviews and ratings that match the string ‘%no vegetarian options%‘:
Average star ratings
The distribution of star ratings is obviously different too:
As you can see on the right, the ground truth is distinctly different. The correlation coefficient between the two distributions on the right is -0.4, while it’s well above 0.9 when comparing any of the three distributions to the left.
So it’s clear that the model has a bias towards positive reviews. In fact, if you look at the baseline distribution from the first 1,000 reviews of restaurants in the ‘American’ category, we can see the underlying distribution that the model was trained on:
Star bias in the data
The new question to answer is what happens to the responses when the training data is balanced for stars? Also, I realize that I need to run a pass through the models with just a ‘review:‘ prompt.
Dammit, the ‘balanced’ training corpora isn’t. Need to fix that and re-train
Bad data
4:15 Meeting
SBIRs
MDA costing meeting
Work on building first pass map. It’s actually working pretty well! Need to write an example script for tomorrow
Create balanced (20k each) star corpora and train – done
Create low star corpora and train (1, 2, 3?)
Installed sentence-transformers, which probably broke sentiment.
SBIRs
Integrate TextComparePopup and try making a map. I’m pretty sure that there will be issues about putting topics into groups and listing topics from different groups – done, and seems to be working well. Tomorrow we try for real?
Spreadsheets for vegetarian 100k GT vs GT vs synth. Everything is good except for ‘no vegetarian options’ It’s the only options that does not appear in the first 100k rows. Going to try some longer prompts to see if I can nudge the model in a better direction. Do that at lunch
Hmmm. I can’t seem to produce a negative star distribution:
viztales 
You must be logged in to post a comment.