2018-2019 Postmortem

2019-07-31

Hello World

As inspired by my friends keur and mdcha, this document serves as a postmortem for the time period encompassing approximately the last year, from Summer 2018 to Summer 2019.

For context and background, this overlaps with my senior year at UC Berkeley, where I was a Computer Science major, and a member of the Open Computing Facility. The mission of the OCF is to ensure that no member of the campus community is without the computing resources they need, and to this end we provide a variety of free technical services such as a 30 seat computer lab, web and application hosting, distributed and high performance computing, and a community of free and open source software and computer enthusiasts who also happen to be my family away from home.

Summer 2018

Internship

In Summer 2018 I was a Production Engineering intern at Facebook in Menlo Park, CA, along with dkessler. I was a member of the PE Operating Systems team, which, among other things, manages the Chef cookbooks that underlie Facebook’s configuration management infrastructure. My project was to build tooling around the configuration and scheduling of builds of certain config management artifacts. I learned a lot about software engineering while working at Facebook, especially in terms of mistakes to avoid in the future. In particular, I realized in retrospect that I could have completed my project much faster if I had done more to explore the ‘risks’ involved in the project before implementing a full solution. During the discovery phase of trying to understand the problem, I should have attempted to write a quick, hacky end-to-end solution to learn which parts of the project were actually complicated, and which I should spend additional time designing, and which parts were immaterial to the success of the project as a whole. This would have also aligned with the principle of “Design it twice” that Professor Ousterhout discusses in chapter 11 of A Philosophy of Software Design. Instead, what happened was that I spent entirely too much time writing beautiful code for the scheduling system, (ultimately unimportant) and didn’t leave myself enough time to properly integrate my changes into the build system (actually mission-critical). It would have been trivial to write the code correctly a second time if I had actually derisked the project properly in a first pass.

Lessons learned: explore risks early to allow you to focus on the problems that actually matter.

I would also like to note my gratitude for the mentorship of Zeal Wierslee, who was my intern manager.

DEF CON

I attended DEF CON 26 with keur, mdcha, dkessler, and wporr. It was my first time going, and it was a lot of fun. We attended a couple talks and visited several villages, went to an EFF roundtable and met several EFF organizers such as nash, went to bcrypt’s tea party, ate good food, and spent a bunch of time hanging out and trying to solve the badge puzzle. While the conference itself was great, the major value for me was going with my friends, for whom I am deeply grateful.

XCF

In Spring 2017 I re-founded the eXperimental Computing Facility. In Summer 2018, we participated in the acquisition of a rack in the Hurricane Electric FMT2 datacenter, and moved some equipment there. It was pretty cool to set up, but nothing particularly exciting happened.

OCF

During summertime, the OCF is not formally in session, but nonetheless we hold planning meetings to talk about the upcoming academic year. We discussed a number of things, including improving diversity in the organization, healthy communication, our onboarding processes, technical projects, reorganizing our administrative structure, future funding, and more.

Fall 2018

School

In Fall 2018, I took CS W186 (Databases), MCB C62 (Drugs and the Brain), and SEA 10A (Introduction to SE Asia). 186 was very interesting - Professor Hellerstein is a great lecturer, and I learned a lot. That the class was all online was a convenient plus. MCBC62 was also a fascinating class. The textbook, Pharmako/Poeia, was even better. I do not have much to say about the quality of SEA 10A.

OCF

In Spring 2018, I figured I would ‘retire’ from leadership in the organization to focus on schoolwork, DeCal, and XCF stuff in the Fall. To the detriment of those other putative obligations, I remained quite involved in management of the organization. One of the major goals in Fall was to prepare for fundraising in Spring 2019, as our STF grant for paying front desk staff was expiring. To that end, I helped present to the ASUC Senate about the history of the OCF, our services, the value we provide to campus, and our ongoing needs. I also participated in deciding which new front desk staff to hire for the following semester. We also worked on improving our relations with the rest of campus, starting by meeting with Professor Scott Shenker to discuss the OCF, a meeting that occurred because keur went to office hours and asked a good question. More details about OCF stuff that happened can be found in the minutes. From a technical perspective, my year-long dream of seeing 10GbE in the lab finally came to fruition with Arista’s donation of a number of (albeit EOL) 10GbE switches to the OCF.

Another fun thing that happened was the Great Outage of Fall 2018. During a regularly scheduled downtime event to restart the hypervisors for kernel updates and to upgrade networking to 10GbE, we ran into signficant difficulty removing the old Cisco Catalyst 2960S-48TS-L from our rack, due to a improperly installed cage nut. In the process of mangling it out of the rack, mdcha accidentally tripped over the PDU’s power cable and the entire rack lost power. We jumped into overdrive to finish replacing the switch, add SATADOMs, replace the NICs, finish the restart and bring the machines back up to preserve our precious 9s of uptime. In retrospect, it was hilarious and a great bonding experience for OCF staff, even if it was somewhat stressful to be in the midst of it.

DeCal

In Fall 2017, I helped restart the Hands-On Linux Systems Administration DeCal. For three semesters I had the privilege of serving as a head facilitator for the DeCal. The DeCal was regularly oversubscribed and we were able to teach over 200 students how to use GNU/Linux from the perspective of a systems administrator. Fall 2018 was my favorite semester, as we not only had good retention and several excellent students who asked engaging questions, but we were finally able to acquire some converts from the DeCal to regular OCF staff.

XCF

As one might expect, nothing significant happened with the XCF in Fall 2018.

Winter 2018

In Winter 2018-2019 I attended six weddings in India, Pakistan, and the US. It was a busy three weeks.

One of the benefits of travelling to somewhere 12+ hours ahead is that, upon my return, the jetlag essentially reset my sleep cycle. I was going to sleep at 10pm and waking up natually at 6am. I was very productive in the morning and felt good the entire day. Needless to say, the moment the next semester started I fell right back into bad sleep habits.

During this time I also started the eXperimental Computing Facility Foundation, a 501(c)(3) designed to externally support the XCF and serve as a vehicle for some of our experimental software engineering efforts. The first thing we did with the XCF Foundation was register AS394119 and acquire IP space for our rack in the HE FMT2 DC.

Spring 2019

My last semester at UC Berkeley.

School

I decided to take only technical classes this semester. I was fortunate enough to get into CS 268, the graduate networking course taught by Professor Sylvia Ratnasamy. 268 was the only class at Berkeley that I actively tried to attend every lecture of. Professor Ratnasamy is an amazing teacher, and the class was utterly fascinating. Taking 268 convinced me to keep my eyes open for possibly pursuing a PhD in computer networking should the future allow for it. I also took CS 184 (Computer Graphics) with several fellow OCFers, including dkessler, gleeb, asai, and jerryzhou. The projects were hard but the collaboration extremely rewarding. For a while I also took CS 152 (Computer Architecture) but I ended up dropping the course and auditing instead as it was a lot of work and dropping it it would allow me to have a semester with no finals. Sorry jkuki. The material, of course, was very cool.

WRCCDC

From Spring 2018 onwards I participated in the Western Regional Collegiate Cyber Defense Competition at the behest of Berke1337, the campus’ “security” club. In Fall 2018 and Spring 2019 I was joined by mdcha and gleeb, along with some other people with Windows experience from Berke1337 to round out the team. With all due respect to Berke1337, the OCF hard-carried our team. As a result of our efforts, we were able to advance from the invitationals to the state-wide competition at Coastline College in Southern California. mdcha, gleeb, and I drove down one weekend early in the semester to participate. Suffice it to say, the competition was hellish. The OCF subteam quickly secured the non-Windows boxes, including the switch and firewall, but our Windows teammates… suffered difficulties doing the same for their machines on the network. The competition ended on day 2 with the Red Team demanding passwords for a single Linux box in exchange for not formatting the drive of the primary Windows domain controller, and over our objections, the password was divulged. To no avail, of course, as the Windows machine was destroyed anyways, along with our chances in the competition.

As with DEF CON, my favorite aspect of this trip was going with my friends - the enhancement of existing friendships and the genesis of new ones. mdcha, being a SoCal native, took us to eat at the correct exclusive places and showed us around various other interesting parts. We spent our days cooped up in the competition room and our nights wondering how difficult it could really be to patch four Windows machines and enable the Windows Defender firewall, and speculating if we might have gotten further in the competition had the OCF formed its own team. (Our conclusion: we’d have wiped the floor with everyone else). A special shoutout is owed to gleeb for single-handedly figuring out how to configure a Palo Alto firewall from scratch on the spot in the middle of the competition.

OCF

At the end of Fall 2018 I decided to abandon retirement and stand for election to be General Manager of the OCF, an endeavor in which I was successful. Spring 2019 turned out to be a critical semester for the organization, and I am enormously grateful for the support of my friends and fellow members in helping to make it a success. To start, the OCF requires funding to pay for operational and capital expenses related to the upkeep of the lab, but more importantly, to pay the salaries of our front desk operations staff, without whom we would be keep the doors of the lab open. This funding was originally provided for two years in the form of a STF Long Term grant for ~$70k, awarded in Spring 2017. We explored several avenues of funding: a student fee referendum, which would directly charge the students $2 per semester to fund the OCF, an extension of our STF grant, an expansion of our ASUC funding, external funding, and more. We spent many hours writing and rewriting the text of the referendum, but it was ultimately rejected by campus as the size of the fee was too small to cover the overhead costs of administering it. We expected this outcome though, and so hedged our bets by following the advice of our former advisor Jenny McNulty and applying for an expanded large-scale grant through STF instead. After many more long hours of writing and rewriting, a process in which many OCFers (special shoutout to keur for leading this effort) came together just hours before the deadline to rapidly bring our grant applications into shape, we submitted two grants, which then condensed into one grant, seeking a total of almost $200,000 in funding over the next two years. We presented about the OCF again at the STF Large Scale Grant Forum. In early May 2019, we got the good news: our grant had been awarded in full! This represents the largest budget the OCF has ever had in its entire 30 year history.

We also made forward progress on several other fronts. For administrative restructuring, we started to trial-run a committees system. We made conscious efforts to make the environment of the organization more welcoming to a diverse group of people. Out of the project to switch OCF from Marathon to Kubernetes came several meetings with IST, and the campus CTO Bill Allison, who offered us help and equipment. Staff meetings began to start with updates from BoD, and OCF staff diversity and retention reached new, and welcome, heights.

DeCal

The DeCal was on hiatus in Spring 2019, because I neglected to file the paperwork in time in the Fall. It was ok though, as I was far too busy in the Spring to have been able do a good job running the DeCal as well. The DeCal will be brought back in Fall 2019.

XCF

No surprises, not much happened with the XCF, as we are blocked on being able to afford a router than can serve full IPv4 and IPv6 tables. Maybe something will happen later.

Alumni Dinner

In late May 2019, after many a week of planning, the OCF held an alumni dinner to celebrate the 30th anniversary of our founding. Almost 50 alumni showed up, from as far back as 1995, to celebrate and eat dinner at China Village restaurant in Albany. The event, I am grateful to say, was an unqualified success and brought many alumni together not only with current staff, but among themselves. Many of them could hardly believe the state of the organization as it stands now compared to when they graduated - but our present successses were invariably built by standing on the shoulders of giants.

Summer 2019

Internship

In Summer 2019, I am interning at a small startup in Berkeley called Kelda that creates developer tools for users of Kubernetes. It has been an enlightening experience so far, and challenging due to the nature and demands of the startup life. I am grateful for the opportunity to be here, for my valued coworkers, the chance to learn about the startup life in a hands-on manner, the technical and product challenges, and the chance to continue to be around campus for one last summer. In particular, I appreciate the nature of the two cofounders, ejj and kklin, especially for their candid and honest feedback, and their willingness to engage and share their own personal stories and theories on careers, goals, relationships, and life. kklin is an excellent verbal communicator: he takes care to speak directly and with conscious intent, and at the end of a conversation, there is no uncertainty or ambiguity in the information or ideas that were exchanged. This is a skill I wish to learn. Both are also composed of focus, and zealously guard their time to spend on high-leverage activities. Another enviable trait.