Author Archives: pgfeldman

Phil 10.1.20

October?? Really??

#COVID

  • Still working on getting Google Translate to work without the os value set (which is misbehaving). This looks to be the answer (from stackoverflow, of course):
# The way I think it should be done
client = language.LanguageServiceClient.from_service_account_json("/path/to/file.json")
# Google seems to want this value set though, for portability across environments?
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="/path/to/file.json"
  • The most recent Google documentation on this requires a storage object, and then doesn’t show how to use it?
# Explicitly use service account credentials by specifying the private key    # file.    storage_client = storage.Client.from_service_account_json('service_account.json')
  • This worked!
import six
from google.cloud import translate_v2 as translate


translate_client = translate.Client.from_service_account_json("path_to_credentials_file.json")
text = u"So let us begin anew--remembering on both sides that civility is not a sign of weakness, and sincerity is always subject to proof. Let us never negotiate out of fear. But let us never fear to negotiate."
target = "de"

if isinstance(text, six.binary_type):
    text = text.decode("utf-8")

# Text can also be a sequence of strings, in which case this method
# will return a sequence of results for each text.
result = translate_client.translate(text, target_language=target)

print(u"Text: {}".format(result["input"]))
print(u"Translation: {}".format(result["translatedText"]))
print(u"Detected source language: {}".format(result["detectedSourceLanguage"]))
  • Results:
Text: So let us begin anew--remembering on both sides that civility is not a sign of weakness, and sincerity is always subject to proof. Let us never negotiate out of fear. But let us never fear to negotiate.

Translation: Beginnen wir also neu - denken wir auf beiden Seiten daran, dass Höflichkeit kein Zeichen von Schwäche ist und Aufrichtigkeit immer einem Beweis unterliegt. Lasst uns niemals aus Angst verhandeln. Aber lasst uns niemals Angst haben zu verhandeln.

Detected source language: en
  • Here’s round-tripping to Arabic:
from google.cloud import translate_v2 as translate


translate_client = translate.Client.from_service_account_json("path_to_credentials_file.json")
text = u"So let us begin anew--remembering on both sides that civility is not a sign of weakness, and sincerity is always subject to proof. Let us never negotiate out of fear. But let us never fear to negotiate."
target = "ar"
source = "en"

# Text can also be a sequence of strings, in which case this method
# will return a sequence of results for each text.
result = translate_client.translate(text, target_language=target)

print(u"Text: {}".format(result["input"]))
print(u"Translation: {}".format(result["translatedText"]))
print(u"Detected source language: {}".format(result["detectedSourceLanguage"]))

text = u"{}".format(result["translatedText"])
result = translate_client.translate(text, target_language=source)

print(u"Text: {}".format(result["input"]))
print(u"Translation: {}".format(result["translatedText"]))
print(u"Detected source language: {}".format(result["detectedSourceLanguage"]))
  • Results:
Text: So let us begin anew--remembering on both sides that civility is not a sign of weakness, and sincerity is always subject to proof. Let us never negotiate out of fear. But let us never fear to negotiate.
Translation: لذلك دعونا نبدأ من جديد - نتذكر على الجانبين أن الكياسة ليست علامة ضعف ، وأن الإخلاص يخضع دائمًا للإثبات. دعونا لا نتفاوض بدافع الخوف. ولكن دعونا لا نخشى للتفاوض.
Detected source language: en

Text: لذلك دعونا نبدأ من جديد - نتذكر على الجانبين أن الكياسة ليست علامة ضعف ، وأن الإخلاص يخضع دائمًا للإثبات. دعونا لا نتفاوض بدافع الخوف. ولكن دعونا لا نخشى للتفاوض.
Translation: So let's start over - remember on both sides that civility is not a sign of weakness, and sincerity is always subject to proof. Let's not negotiate out of fear. But let's not be afraid to negotiate.
Detected source language: ar

GOES

  • More working through the algorithm. I want to make plots for the normal (rotation) vector to see how that looks. That could also be plotted in 3D. Hmmm.
  • So that’s done, and it’s not as smooth as it should be. Here’s the reference frame rotated through 180 degrees on the left, the matching rotation in the middle, and the simplified reaction wheels with the rotation axis (cyan) on the right:
  • That cyan plot really bothers me. The results aren’t bad (vehicle), but I wonder if it’s because there are two choices that are equally good and it’s alternating between them? Let’s print out the name of the axis and the runner’s up (sorted by angle):
x angle = -1.00, y angle = -1.00, z angle = -1.00
z angle = 10.05, x angle = 10.03, y angle = 1.41
x angle = 10.11, z angle = 10.05, y angle = 2.02
z angle = 10.27, x angle = 9.95, y angle = 2.69
z angle = 10.03, x angle = 9.97, y angle = 1.87
z angle = 10.02, x angle = 9.96, y angle = 2.10
z angle = 10.00, x angle = 9.92, y angle = 2.25
z angle = 9.99, x angle = 9.85, y angle = 2.32
z angle = 9.96, x angle = 9.76, y angle = 2.34
z angle = 9.94, x angle = 9.66, y angle = 2.40
z angle = 9.91, x angle = 9.57, y angle = 2.59
z angle = 9.88, x angle = 9.54, y angle = 3.00
z angle = 9.85, x angle = 9.61, y angle = 3.67
x angle = 9.81, z angle = 9.81, y angle = 4.60
z angle = 11.31, x angle = 9.32, y angle = 6.58
z angle = 9.73, x angle = 9.45, y angle = 4.32
x angle = 9.90, z angle = 9.69, y angle = 5.66
z angle = 11.64, x angle = 9.10, y angle = 7.57
  • The x and z axis are almost identical. Would it make sense to average the closest? Let’s try something more extreme:
x angle = -1.00, y angle = -1.00, z angle = -1.00
y angle = 14.11, z angle = 14.11, x angle = 12.82
z angle = 20.31, x angle = 17.04, y angle = 14.19
y angle = 17.53, z angle = 13.48, x angle = 13.01
z angle = 14.25, y angle = 13.54, x angle = 6.81
y angle = 13.78, z angle = 12.22, x angle = 6.76
z angle = 12.27, y angle = 10.56, x angle = 6.80
z angle = 10.82, y angle = 8.63, x angle = 7.64
z angle = 10.29, x angle = 9.36, y angle = 6.14
x angle = 10.28, z angle = 10.00, y angle = 3.70
z angle = 9.85, x angle = 9.85, y angle = 3.70
z angle = 10.29, x angle = 8.94, y angle = 5.09
z angle = 10.82, y angle = 8.09, x angle = 7.32
z angle = 11.50, y angle = 10.74, x angle = 5.69
y angle = 13.27, z angle = 12.22, x angle = 6.12
y angle = 13.54, z angle = 13.02, x angle = 4.16
y angle = 14.09, z angle = 13.21, x angle = 6.16
z angle = 15.23, y angle = 14.19, x angle = 12.12
  • 10:00 Meeting with Vadim. Went over the code and found that the angle calculation only works properly between two unit (vectors of the same length?). That fixed the contribution problems I was having. So that’s one serious bug fixed. Vadim is going to look at folding in the changes, and I’m going to work on getting this to work with the six reaction wheel version.
https://viztales.files.wordpress.com/2020/10/image-2.png
  • Status report

NESDIS

  • Finish paperwork! Done

Phil 9.30.20

Discovered State Azure, which is some very nice chill music

My thoughts on the debate last night:

https://twitter.com/philfeld/status/1311245626873708550

#COVID

  • Creating a list of distinct content that translated to “elderman”. Going to see if I can get the Google Translate API to deal with these problem children
  • Installing the python libraries for google translate
  • Because I love pain, upgraded tensorflow. Let’s see if anything still works! It does! At least for translation and GPT, which is good enough for me at the moment
  • Working on getting the translate API running

GPT-2 Agents

  • The paper’s submitted!
  • Had a good chat with Shimei and Sim last night. The db has been uploaded, and we talked about next steps. I also showed how to install the Huggingface transformers library from source. That involved uninstalling the Typing library for some reason. Seems there are conflicts?

GOES

  • 10:00 Meeting with Vadim & Erik
  • 2:00 Status report

Phil 9.29.20

Behind Twitter’s biased AI cropping and how to fix it.

In Isolating Times, Can Robo-Pets Provide Comfort?

COVID

  • Updated my DaysToZero code to put charts into the spreadsheet, which was pretty straightforward using xlsxwriter. There do seem to be three basic patterns:
  • The first is steady growth:
  • So no curve flattening here. The disease is moving pretty steadily through the population. The USA is mostly on this track as are countries like India and Chile. I think the difference that we see is related to a first, faster wave among the more vulnerable populations.
  • The second pattern is the ‘flattened curve’. Ireland shows this really well, as does New York state:
  • The last pattern is the ‘second wave’ pattern. Japan seems to be having one now:
  • So it looks like we are far from out of the woods on this, and letting your guard down is soundly punished.

#COVID

  • Start the db to fixing “elderman” posts. Running. I’ve got about 100k bad posts. The fixes seem to be taking care of some. This is going to require multiple passes

GOES

  • 2:00 Meeting with Vadim

GPT-2 Agents

  • Updated the document. Antonio’s going to submit. Fingers crossed! It would be nice to go to London in May
  • 3:30 meeting?
  • Need to backup and save the db. Done. It’s almost 11GB! Compressing. Backed up the compressed version, which is (only!) 3GB.

Phil 9.28.20

GPT-2 Agents

  • Working on paper – done for now, off to Antonio
  • The data extraction finished! There are 9,375,587 ingested tweets
  • Need to backup and store the db

GOES

  • GVSETS rehearsal at 3:30 – recorded!
  • 2:00 Meeting with Vadim

NESDIS

  • Finish paperwork

Book

  • Finish transcription

Phil 9.25.20

Call Competitive Cyclist about this

Book

  • Adding to these thoughts about using three stories to frame nomad, flock, and stampede behaviors.
  • Listening to BBC Business Daily on London’s dirty financial secrets. In the episode, Tom Burgis, author of a new book Kleptopia: How Dirty Money is Conquering the World, discusses money laundering. It’s making me think about how though money is a dimension reduction process, it’s a very peculiar one. The ability to simplify transactions and “store” the profits means that power accumulates with money. That money lets the owner of the money pay people to add dimensions in other areas. This can be good, as with scientific research, or it can be bad, as with the creation of the byzantine dimensions of money laundering. In the middle somewhere are activities like high-speed trading.
  • The thing is, any increase in the ability to create social realities that exist independently of the environmental reality creates the conditions for stampedes. A lot of crime (Scams, cons, embezzlement) depends on the creation of a social reality that overwhelms trustworthy information coming in through other channels.
  • More transcription

GPT-2 Agents

  • At 7.5M tweets so far
  • Got Antonio’s comments back. Need to roll them in

GOES

  • More rotations. I think we found the problems. The first is that the angles being fed to the absolute angle calculations were wrong. The second was that the sign of the pitch and yaw vectors flip near 180 degrees and we were not compensating for that.
  • Need to try a runthrough for the GVSETS slides. Too tired. Maybe over the weekend.

NESDIS

  • Gobs of paperwork

Phil 9.24.20

Fireplace (410) 203‐2876 Invoice No. 1837 Issued on Tue Jan 21, 2020

Book

GPT-2 Agents

  • About 6.5M tweets processed so far
  • 3:30 Meeting

#COVID

  • Still reading in labeled tweets – done!
  • Need to fix “elderman” translations, though there are some other bad/partial translations as well

GOES

  • Create logger for DDict – done!
  • Start rwheel coding for incremental rotations – started
  • 2:00 Meeting and demo. It went well, I think. We can run 100x speedup now!
  • Vadim has this thought: Was just thinking that at the next demo meeting, we should mention that this is adaptable to many different situations, including simulating the Launch Orbit Raising scenarios, specifically for the upcoming GOES-T launch. Since it’s a physics sim and we can make the various pieces move, such as deploying the solar panels.

JuryRoom

  • 5:30 Meeting

Phil 9.23.20

Fireplace (410) 203‐2876 Invoice No. 1837 Issued on Tue Jan 21, 2020

MD Food bank

Book

#COVID

  • Adding in the labeled values to the Arabic data
  • Fix “elderman” translations

GOES

  • 10:00 Meeting with Vadim. Worked through bugs in the angle adjustment code and added clamping. Still have problems at 120 degrees. Need to see if this is a blocker for the demo
    • Roll freezes at exactly 90 degrees. WTaF? Need to extend the DDict so that it can dump a column-format csv file that is incrementally updated
    • Pitch works great. It’s neat to see how all the rwheels line up for that maneuver
  • 12:00 All hands
  • 2:00 Status meeting

Phil 9.22.29

https://public.flourish.studio/visualisation/3603910/

Today the USA passed 200,000 dead from COVID-19. That triggered a memory from the earlier days of the pandemic. Italy handled the virus poorly, and I remember thinking that will be the standard that the USA will be judged. It should have been relatively easy to do better than Italy.

In the chart above, I scaled the deaths for selected countries so that they could be compared to the US directly. As you can see, we are now worse than any country in the EU, and staggeringly worse than South Korea.

Not sure what to do about that other than just be angry.

Book

  • Transcribing
  • Reading more on the money book, and coming to the conclusion that money provides consistent mathematical rules for transactions that promote coordination

GOES

  • 10:00 Meeting with Vadim. Walked through a lot and fixed a few things. There seems to be some problem related to 120 degrees, which is the angular spacing of the reaction wheels. I think that we are hitting a singularity. Working on a short term fix, though I think the long term is to simply use the reference frame code I’m working on
  • Speaking of which, I need to have a case for handling overlapping vectors (angle = 0), and not using an angle that is too small, and using the last angle if possible. If no angles are available, then wait until the next time and get larger angles? Done!

#COVID

  • Processing labeled tweets

Phil 9.21.20

Welp, I missed my 3 day weekend getaway. I thought it was next week. Perfect weather too. Stupid.

Book

  • Continued transcribing

GOES

  • Going to try and see which vector sweeps out the largest arc and use the cross product for the vector to rotate all the objects in the frame.
  • That worked!
  • It’s a little rough occasionally, but it tracks well

GPT-2 Agents. Somewhere north of 4M tweets ingested

Phil 9.18.20

Get deposit!

Opened a ticket for VPN access issues

Book

  • Scanned notes
  • Playing interview into Google Docs, where it is doing a reasonable job of transcription!
  • Meeting with Michelle. Went over the text a bit and decided to skip next week

GPT-2 Agents

  • Progress as of this morning:

GOES

  • Continue on incremental rotations. Here’s the reference and vehicle both using absolute angles
  • Now, here’s the reference using incremental and the vehicle using absolute
  • This is why we need to calculate the plane of rotation each time of we’re using incremental rotations.
  • Thinking about this some more. I can check to see which vector sweeps out the largest arc and use the cross product for the vector to rotate all the objects in the frame.
  • 10:00 meeting with Vadim. There’s an issue where the calculated angles loop from +180 to -180. Here’s the code I use to catch that. It’s dumb, but I can’t find better:
def closest_angle(a1, a2) -> float:
diff = a1 - a2
if diff < 0:
while abs(diff) < 180:
diff += 360
return diff
while abs(diff) > 180:
diff -= 360
print("{} + {} = {}".format(a1, a2, diff))
return diff

Phil 9.17.20

GPT-2 Agents

  • Almost 2,000,000 tweets ingested as of this morning

#COVID

  • Work on fixing all the text broken by “@xxxxx”
  • Ingest the rest of the labeled data

GOES

  • 10:00 Meeting with Vadim

Book:

  • 2:00 Meeting with Ted Hiebert, hopefully. Got the questions written and printed out. Tested the voice-to-text in Docs, which is extremely good.
  • Interview went well

ML group Meeting at 3:30

JuryRoom Meeting at 5:30

Phil 9.16.20

Finishing a review for Shimei. Done!

IRAD

  • Make a smaller slide deck – done

GOES

  • 10:00 Meeting with Vadim We’ve still got this oscellation problem. Going to try and use simplified rotations to see if that exposes the problem

#COVID

  • 3:00 Meeting – went over translation progress. Need to write a regex to remove “@xxxxxx “

6:30 Dinner for Gary

Phil 9.15.20

#COVID

  • Translate CSV file and insert into db
  • My desktop box is so slammed that it can’t do anything else, moving this over to the laptop
  • Moved project to svn because github won’t let you store large files like models and databases
  • Getting the dev environment set up. Lost pip somehow – reinstalled
  • Got the model localized. Here’s the code to do it (from here):
from transformers.file_utils import hf_bucket_url, cached_path

src = 'ar'  # source language
trg = 'en'  # target language
pretrained_model_name = f'Helsinki-NLP/opus-mt-{src}-{trg}'
archive_file = hf_bucket_url(
    pretrained_model_name,
    filename='pytorch_model.bin',
    use_cdn=True,
)
resolved_archive_file = cached_path(archive_file)

print(resolved_archive_file)
  • Added a “label” column, and inserted the labels and translations
  • Need to back up the db
  • Sent the csv with translations out to the group

GPT-2 Agents

  • Postponed meeting
  • At over 700k ingested Tweets

GOES

  • 2:00 Meeting with Vadim

Book

  • Set up interview – done! Set up for 2:00 this Thursday

Phil 9.14.20

GPT-2 Agents

  • Added Antonio’s suggestions and uploaded the paper to Overleaf. Looks like about one more page to play with
  • Working on Twitter parser – done-ish?
  • Wrapped the parser with a glob object and started the full run. 10,000 processed tweets and climbing!

Marketing:

  • DATC Working group meeting and presentation.
  • Presentation was technical as specified, and sailed over the heads of everyone, I think

GOES

  • Postponed the meeting with Vadim because I thought the workshop ran all day

Phil 9.11.20

I would love to see a timeline of all the things that we’ve done in response to 9-11 and how they’ve worked out

GPT-2 Agents

  • Parse timestamps – done
  • Hand-annotate the schema?
    • datetime – done!
    • arrayobject
  • Start running analysis? Wrote a tweet to the db. Calling it a day
  • 2:30 Meeting with Shimei and Sim – started a google doc for tasking

GOES

  • 10:00 Meeting with Vadim. I think the goal should be to tune the rwheels so that the yaw flip curves start to look more realistic, and then see how it works at speed. One of the issues that we need to think about is the role of mass in high-timestep physics. Would lower mass make better behavior?

Book

  • 2:00 Meeting with Michelle – tweaked the dimension reduction section
  • Write email to Theodore Hiebert – done
  • Looked a bit into voice recording and transcription. The best answer still seems to be Google? Here are other options