Meetups/Infra/2025-11-03

brief discussion of zfs (and 3rd party web gui via cockpit), demo of Apache Ray for ML inference pipelines, critique of Slack -- time cost -- hopeful love of zulip, AWS outage talkthrough

Introductions

• [name] - [background]. [goals for meetup, or interests to explore]

• Loren --
• Eugene -- filmmaker, vr films, occulus in irvine. Interested in taking control of my data. Reliable control of data.
• Frank -- software bg, interest in home labbing
• Zoe Radavisch (sp) -- dayjob moneyball stuff for phillies, platform and devops
• Patrick -- platform & devops, interested in homelab
• Gwen -- here to learn
• Derek -- programmer, getting into LEAN4
• Greg -- programmer, interest in computer infra broadly speaking
• Robert -- no particular interest...yet
• Conner -- aerospace engineer, work on lots of small computers, here to broaden horizones,
• Matt -- used to do sysadmin stuff with Loren as student at Berkeley, no longer work in it, down to talk about k8s
• Alex -- do ML stuff for work, now do some for fun, happy
• Lulu -- came here to learn
• Doug -- like self-hosting, lots of data
• Amia (sp) -- working on hardware & infra, getting more into embedded systems
• Max -- repeat offender to infra, read about nix stuff
• Erik -- like hardware stuff. Recent k8s setup, and network setup.

Lesson or Demo

• Read aloud: clarify for meetup. We are taking notes in a riseup pad (or I am--help appreciated, and links). We have meeting notes posted to the wiki. noisebridge.net, search Infra, or Meetups/Infra. (the Infrastructure page has a disambiguation link.)
• Shell, web services, self-hosting, networking!

• zfs -- try cockpit, for zfs data stuff.
  • home much data are people managing

100 TB
30 TB, but lots of files, lightweight

syncthing
dropbox -- growing frustration and cost.  selective sync, getting worse.
how to you keep the 3-2-1 system going.
backblaze.
3-2-1 NAS system. 3 types of backups, 2 types of devices (physical), 1 cloud backup
have the hardware? yes. why cloud?

• network stuff at noisebridge --

• Greg used to work at Cisco Meraki (enterprise cloud-managed networking), on the web UI - e.g. https://meraki.cisco.com/products/meraki-dashboard/ - although that was many years ago and they have updated it somewhat since then

• Alex, Ray -- apache distributed. https://docs.ray.io

I'm more a user than operator. I'm an ML scientist.
Ray provides framework, training pipeline.
With spark you can do lots of complex joins, a big ETL pipeline
Ray provides good last-mile api/framework for feeding data into models
Set up works, 
Scale with 
specify cluster configuration in yaml, invoke, resource needs in python, decorators on functions
Examples of great usage:  serving live inference, nice to have on k8s. Use a lot for training & batch inference--assemble data throughout the data, run.
For hyperparameter tuning--give ranges? with HyperOpt integration, will adaptively choose good next candidates--explore/exploit.  Doug: github library Gaussian 

Fun uses for Noisebridge?   Doing bsky processing. 
Embedder app. Might be able to help us coordinate across jobs.
Kubernetes Operator
Bsky embedder -- Using CLIP to classify images
Ray -- supports checkpointing?  no, you do yourself. Maybe awesome-ray,https://github.com/JiahaoYao/awesome-ray 

• Zoe, slack message eval

"high blood pressure is the silent killer" -- think slack is that for orgs. Constant interrupts so bothersome.
At prev co, all chanenls were public by default.
Did a back of the napkin calc, if it taks 3s to read a slack message--and counted the messages across all a company.
Isn't that a CIA strategem? https://www.cia.gov/static/5c875f3ec660e092cf893f60b4a288df/SimpleSabotage.pdf
In chatting with Loren before meeting--observation that it's always too much or too little.
Has anyone seen good -- at organization?
No -- it's the territory of orgs. Empires. Secrets
Was addicted at prev org. Now on discord here.

Paul Graham essay, work all night. Meeting people vs.  Maker time vs manager time.

metcalf's lawhttps://en.wikipedia.org/wiki/Metcalfe%27s_law 
in same vein, "microservices aren't a programming tool, they're an organizational tool


AWS -- scaling effectively. 2 pizza teams. 
IRA terrorist cells. (6-) 9 people. 1 reported up. 1 controlled down.
Similar number in meetings.

In social gatherings, 3-4 people stable. 5 into 3-ad, 2-ad. 
Had 3 times of day to check slack. Morning, noon, night.
Great book about this from the 80s, People-ware. https://www.amazon.com/Peopleware-Productive-Projects-Teams-3rd/dp/0321934113#detailBullets_feature_div
 quiet offices
 frequency of paging throughout day
 -> make orgs successful, how able are individuals to cordon off attention, time, from interruptions

IETF coordination.  Between IEEE and W3C (WWW consortium) IETF RFCs can be searched : https://www.rfc-editor.org/errata.php 
Mythical Man month -- optimal team size -- at 5, or less than 5. At 5, team. 
Linux -- 
things grow and break into a mess
Slack -- tool for bad game co

https://oxide-and-friends.transistor.fm/ - recent book reviews podcast

• AWS outage -- Matt & Loren

route53 is their dns service.
plan & executor
TOCTOU check

https://aws.amazon.com/message/101925/

Outros

Ray was cool
Now remember how much I miss my cube -- 
  gather.town
  discord voice channels, like this, but less skeuomorphic
  have seen other services like this
Urban Myth at Mars -- got a carpet tile there every 5 years you were there.  You couldn't step on someone else's tile. Middle of office more important
Interesting, want convo about BMCs, such things. IPMI. Intel vPro [something]
Fascinated -- local inference, non-cuda based arches. Local AI [schticht hala chip?] (sp). Non-cuda.
GPD Win Mini 2025 (HK-based mfgr, steamdeck competitor).https://gpdstore.net/product/gpd-win-mini-2025/ AMD chip. Basite -- fedora base.
Bamboo printers help.
ssh -- ssh agents, port forwarding. jump. control socket. keys. 
learned about AWS outage
More LEAN4

Questions, Discussion, or Coworking

• [Issue]

For next time

Readings & Exercises

• Readings
• Exercises

Join online

• Try it yourself!
  • Join libera.chat #nb-meetup-infra

https://www.noisebridge.net/wiki/Meetups/Infra