Fork me on GitHub
Jess Meyer - IT Revolution (she/her)18:10:38

Welcome our next speakers @genek101 and @stephen!


Hello! @stephen and I are so happy to present the results of our second year studying software supply chains, with our friends from Sonatype — it is such an amazing data set to be able to study!!!

Stephen Magill [Sonatype]18:10:59

Hi! Great to be hear @genek101 and excited to share these findings with everyone!

Stephen Magill [Sonatype]18:10:26

The report is available here: if you want to follow along!

Stephen Magill [Sonatype]18:10:59

(we’ll mostly talk about Chapter 3 and 4)


Of course, those friends are the folks at Sonatype, who runs Maven Central, which every Java programmer and anyone who runs on the JVM benefits from every day!

Stephen Magill [Sonatype]18:10:28

We all know @genek101 ❤️ Clojure 🙂

😄 1
😂 1
Stephen Magill [Sonatype]18:10:44

And we all ❤️ the JVM I image 🙂


The Haskell bar is so short, it didn’t actually render on my screen, @stephen! 😆


It was super cool to look at a different aspect of software supply chain — last year were the components. This year is the consumer side of equation!

Stephen Magill [Sonatype]18:10:43

Bonus fact: About 1/3 of projects in Maven Central were not part of the “software supply chain”. They were isolated projects (not used by anyone and not using open source libraries themselves).


Faster is better.

Myles [Sooner Safer Happier]18:10:03

"Projects that release frequently have better outcomes and are more secure" - Sooner, Safer and Happier are interrelated and co-dependant.

Roman Zillek18:10:09

Is a fast changing UX and customer journey through applications not leading to end user frustration ?


HI @bryan.finster!!!

Bryan Finster - Walmart (Speaker)18:10:11

I just ran create-react-app. 1625 packages for "hello world". Doesn't include transitive dependencies.

😂 3

Bitcoin miners!!!


OMG, @bryan.finster. It’s so sad…

Stephen Magill [Sonatype]18:10:38

npm has taken dependencies to a new level!

Billy Jo (he/him)18:10:50


👆 1

Try react-native!

Camilo Piedrahita - Bancolombia - IT Manager18:10:59

when you're using JVM o .net dependencies its easy. what about flutter, Erlang and other "strange" dependencies? @stephen

Adam Hawkins, SRE at Skillshare, Host18:10:18

do not run snyk against a react app. It will take a long time 😞


I heard react-native 100x worse. And MUCH faster moving. I heard, “if you don’t update dependencies for 2 months, you’re basically sunk. You’ll spend a week getting builds going again.” 😱😱😱

👍 1
Stephen Magill [Sonatype]18:10:56

good question, @capiedra! In more niche languages you usually have less choice of libraries but I find there are also typically shorter dependency chains — smaller community = more standardization around libraries.

Stephen Magill [Sonatype]18:10:56

(this is anecdotal — I haven’t looked at data here)

Myles [Sooner Safer Happier]18:10:34

"Projects with more dependencies stay more up to date" - super interesting result


I think the first couple of times we ran the analysis, we thought we had “reversed the polarity” and gotten it backwards. Remember that, @stephen? 😂😂😂

Stephen Magill [Sonatype]19:10:25

yes! same thing happened with a question about “how many internal forks of open source projects do you maintain?” the companies most involved in open source scored the highest! we were meaning to ask about long-lived forks that diverge from upstream and thought we had the polarity wrong, then realized that — of course! — to contribute changes back to an OSS project, you have to fork it :face_palm:

Brandon Brown - Nike - Sr. SW Engineer II18:10:01

is the result of hypothesis 3 due to accumulated network effects?

Stephen Magill [Sonatype]18:10:53

Could be for sure. larger teams, larger networks, more code gets pulled in.

Rob Cuddy - DevSecOps Evangelist18:10:29

Does popularity make things resistant to change? Maybe that is why the hesitancy to update

Brandon Brown - Nike - Sr. SW Engineer II18:10:03

I look at time and frequency of updates to determine if i want to incorporate a package.

💯 1
Stephen Magill [Sonatype]18:10:19

That’s a great practice!


^^ @robert.cuddy Actually, the most popular projects have highest release frequency. hang on. Looking for graph…


Note that the most popular projects (upper left) tend to have most freq updates!

Stephen Magill [Sonatype]18:10:21

Note the missing dots in the lower left there.

Brandon Brown - Nike - Sr. SW Engineer II18:10:38

I'd love to see vulnerability spread (contagion) on this dataset on a per-language basis.

Stephen Magill [Sonatype]18:10:15

Vulnerability spread meaning how they propagate through transitive dependency chains?

Stephen Magill [Sonatype]18:10:39

That’s the “fast release” zone — they’re all more popular than the rest.

Rob Cuddy - DevSecOps Evangelist18:10:32

Thank you @genek101 - was trying to correlate that to the comments around Hypothesis 4 and the comments around updates. So is the real issue there changing the project but not paying enough attention to the dependancies underneath?

Stephen Magill [Sonatype]18:10:58

That’s right, the blue dots over on the left of that diagram are staying up to date as they release. The dots that aren’t blue on the left are releasing frequently but not using that release velocity to keep dependencies up to date.

👍 1

I love these diagrams! Migration patterns between versions!!!


I remember the flight from PDX to Orlando working on getting these “arc diagrams” working in Vega.

Rob Cuddy - DevSecOps Evangelist18:10:02

So... that gap between the humps... is that due to your connection in DFW? :rolling_on_the_floor_laughing:

Ben Williams - Arvest Bank - Sr Data Pipeline Dev18:10:39

My leadership still doesn't like going to x.0 versions. They want to go to x.1 versions, like those versions don't have new and different bugs.

Bryan Finster - Walmart (Speaker)18:10:01

Or that the number means anything

Stephen Magill [Sonatype]18:10:17

Love this point, @bryan.finster! We had to throw out more data than I expected to because so many projects don’t even follow standard versioning practices.

Bryan Finster - Walmart (Speaker)18:10:01

Why do people waste creativity on things not related to functon?


For those who want to make an arc diagram in Vega, here’s my simple example:

👍 1
Ben Williams - Arvest Bank - Sr Data Pipeline Dev18:10:53

I like the http://YYYY.MM way to version because it shows it is meaningless.

Ben Williams - Arvest Bank - Sr Data Pipeline Dev18:10:35

I know you will need more points in that method Bryan...


Holy cow. Making those diagrams made me realize how bad people are at following good version numbers!! cc @stephen

Daniel Cahill - Engineer - Ontario Systems18:10:15

I wonder how all of this affects the barrier for entry to try new packages and understand well enough to keep building new features.


Had to write a ton of special cases to convert version numbers so they could be sorted.

James Chen-Smith (Walmart)18:10:48

Re: versioning, Just stick with 😋

🤯 3
😂 1

^^^ @bwilliams4 Totally agree!!! (Because semantic versioning, as we all know, is pretty much useless.)

Bryan Finster - Walmart (Speaker)18:10:04

Is that like the VanillaJS framework?

😅 1
Luke Rettig - Target18:10:23

thats where my head was going

😂 1
Luke Rettig - Target18:10:59

same wavelength

💯 1
Luke Rettig - Target19:10:28

Gene just entered the wavelength

Stephen Magill [Sonatype]18:10:15

“DevOps Padawans” — keeping up the Star Wars references… 🙂

👍 2
Myles [Sooner Safer Happier]18:10:41

@stephen could you post the chart in channel?

Michael Baca, Development Manager, American Airlines19:10:07

What are some good resources to start building towards the High Performers - DevSecOps teams?

Brad Kirchman19:10:10

This is really neet, but not very Colorblind Friendly 😞

Scott Bullitt Thompson19:10:13

What are the the actual factors or ingredients which "push" DevSecOps?


@brad.kirchmann ^^ wow, @stephen just made a high contrast version for you!!! Nice!!!

🙌 1
Stephen Magill [Sonatype]19:10:52

These are the main differentiating factors, so good places to start in transforming practices, @scott.2.thompson and @michael.baca.

👍 2
Michael Baca, Development Manager, American Airlines19:10:27

Paying focused attention right now....😉

Michael Baca, Development Manager, American Airlines19:10:23

Did you see any distinction in terms of the centralized tools used to scan artifacts? Were some tools better than others?

Scott Bullitt Thompson19:10:16

Thanks @stephen. I'm highly encouraged that I'm on the right track! We have Jenkins script integrating with Veracode and Blackduck with thresholds on fails or passes.

Stephen Magill [Sonatype]19:10:20

Those are great practices, @scott.2.thompson. @michael.baca, we didn’t ask about or compare specific tools. Just whether tools from particular classes (SCA, SAST, etc.) were used.

👍 1
Rob Cuddy - DevSecOps Evangelist19:10:26

"Security being integrated into developers daily work" - @genek101 Making security a natural part of what developers are already doing is paramount!

Stephen Magill [Sonatype]19:10:45

More reason to limit and carefully choose dependencies. developer flexibility.


Bifocals not strong enough to see the iPad Pro I was using as a monitor!!!

🤓 1
🙏 2
Chris Gallivan, FCA, Builder of JOY19:10:52

Make security a part of development

Stephen Magill [Sonatype]19:10:52

Curious: Does that resonate with anyone?

Bryan Finster - Walmart (Speaker)19:10:19

To a point. The devil is in the details.

Stephen Magill [Sonatype]19:10:54

Yeah, sometimes a new library is just the right fit and more than makes up for the added complexity.

Bryan Finster - Walmart (Speaker)19:10:26

I've seen leadership run with this idea and take it to "all teams will use exactly the same tools." Variance adds costs and standardization can inhibit improvement. Need balance.

Bryan Finster - Walmart (Speaker)19:10:10

I'm thinking about my context though. > 2000 teams

Stephen Magill [Sonatype]19:10:26

oh yeah, standardizing across so many teams seems destined for failure. definitely a balance.

Bryan Finster - Walmart (Speaker)19:10:42

I think driving standards on dependencies for things like the website is a win though. Reigning in some of the cowboy there would really help.

Stephen Magill [Sonatype]19:10:37

You don’t like my favorite library for formatting dates? I wrote it myself because every other date / time library gets the abstractions all wrong…

Bryan Finster - Walmart (Speaker)19:10:07

Worse. "I didn't bother to look if we had an application already for this solution, so let me demo this thing we are releasing next week to my area" that duplicates 4 others and should be in Platform's domain anyway.

Bryan Finster - Walmart (Speaker)19:10:48

Constant cat herding of bespoke platform teams.

Bryan Finster - Walmart (Speaker)20:10:37

It's getting better rapidly. Proud to say that I'm helping that improve by ignoring whatever "scope" I'm supposed to have. My job is to help teams discover how to deliver better. Anything the constrains that is in my wheelhouse, as far as I'm concerned.

Daniel Lemire19:10:12

I'd love to understand better what is meant by "enforce gov policies in CI"

Matt Cobby (Director of Engineering, Deloitte)19:10:40

I've seen examples of stopping pipeline if certain level of CVEs found or Open Source licences that are not approved. We've built a compliance gatekeeper that won't allow deployment unless you have certain mandatory checks attestations.

Stephen Magill [Sonatype]19:10:19

yes, it’s these sorts of practices

Stephen Magill [Sonatype]19:10:44

in CI = as part of dev / build / test. we also asked about centralized scanning outside of dev, but within CI was more effective.

Ben Williams - Arvest Bank - Sr Data Pipeline Dev19:10:41

The closer the effort is to prod, the more expensive it is to fix. Security has to be part of design not just develop/code.

💯 1
Myles [Sooner Safer Happier]19:10:08

I agree. I have documented some patterns for how that can work in "Sooner Safer Happier" (book release 10th Nov) . These have been proven at scale in a large bank.

Bryan Finster - Walmart (Speaker)19:10:10

Yes, I want to know on my desktop. Anything after that is increasingly less optimal

💯 1

PS: @robert.cuddy Some amazing experience reports showing this on plenary stage. Tomorrow morning, Dwayne Holmes from a large hotel company, and GitHub upgrading from Rails 2 to Rails 5 @eileencodes (closing Keynote). Such amazing talks showing how people operationalize this! (And the consequences of not doing so!!)

🙌 1
Fred Ghahramani19:10:47

love the contour map and interesting on High Performance beating security first.

Dave Fugleberg19:10:35

'beating' if dev productivity is more important...lagging if risk management is the priority. All depends on what you prioritize. In any event, they are complementary, not competing.

Luke Rettig - Target19:10:51

OSS alignment sounds so Zen. i’m feeling this

Jess Meyer - IT Revolution (she/her)19:10:22

Welcome our next speakers @arun.infy and @useidel!

Roman Pickl - technical pm - Elektrobit19:10:57

Love the graphs. So many ideas how to use them in a related context (e.g. update behaviour of customers)

Ben Williams - Arvest Bank - Sr Data Pipeline Dev19:10:07

I use Newtonsoft JSON and CsvHelper alot because I don't have to be an expert in coding that functionality. If you were not already sold in on OSS.


THANK YOU! will research these soon!!!


Yesterday, I read about an amazing .NET library for parsing durations and times that I was jealous of.

Ben Williams - Arvest Bank - Sr Data Pipeline Dev19:10:37

I need to know the name of that one, time is a hard problem.

Chris Gallivan, FCA, Builder of JOY19:10:39

Any fav scan tools for detecting vulnerabilities in open source ?

Scott Bullitt Thompson19:10:37

Wouldn't call it a favorite, but I use BlackDuck. It works, but lacks good reporting, but our Infosec dept likes it. :)

Stephen Magill [Sonatype]19:10:17

@chris.gallivan: Sonatype for analysis of open source and Muse for analysis of your code (see our vendor dome talk tomorrow 🙂 ( and

Rob Cuddy - DevSecOps Evangelist19:10:47

Chris I would suggest visiting the sponsor channel and checking out the vendors there as well. Lots of great choices.


That upcoming talk from @eileencodes is UNFREAKINGBELIEVABLE!!!! Last talk on Day 3.

❤️ 1
Roman Pickl - technical pm - Elektrobit19:10:05

I loved the one you shared on Twitter a few weeks ago.


Upcoming one is even better, IMHO, because she added a section directed at tech leaders. So good!!! It’s so powerful.

❤️ 1
👍 1
Ron S19:10:17

@chris.gallivan We use BlackDuck. It does vulnerabilities and license checking

Jon Sturdevant - Tech Advisor - BlueCross BlueShield of SC19:10:24

One of the characteristics of open source components we've really put a lot more work into has been licenses. There are tons of licenses out there and not all of them are enterprise friendly



thankyou 2
Nick Eggleston, IT Architect (open to offers)19:10:33

@genek101 nice plug for @eileencodes's talk later this week

Daniel Cahill - Engineer - Ontario Systems19:10:52

@stephen I was hoping for your daughter to wave again!

Stephen Magill [Sonatype]19:10:21

Haha, thanks! I’ll have to rope in one of the other kids next time so they don’t feel left out!

Bryan Finster - Walmart (Speaker)19:10:02

Thanks @genek101 @stephen always terrifying to see this material.

Stephen Magill [Sonatype]19:10:57

Thanks! Love digging into this stuff and sharing what we learned!

Blake (Community at DZone; Gene Kim Fan Club Member 2,000,005)19:10:02

Hi @stephen - I was curious if you could describe the differences between Dijkstra’s, A*, and Jump point path finding algorithms?

Stephen Magill [Sonatype]19:10:47

@blakee! Good to see you! Actually, there are similarities between A* and various program analysis techniques like symbolic execution or static analysis. The jump point optimization is similar to optimizations in graph traversal that are used in program analysis and model checking. Not sure if you were serious though 🙂

Stephen Magill [Sonatype]19:10:55

And A* is typically used in online planning whereas Dijkstra’s is more common in off-line / batch contexts (I believe) — but I’m not a planning / optimization expert 🙂

Stephen Magill [Sonatype]19:10:49

np. I have to give a tongue-in-cheek answer too given the topic of the talk with @genek101. The difference is… 3 dependencies vs. 0 🙂



Nick Eggleston, IT Architect (open to offers)19:10:41

@genek101 it would sure be helpful to have a buffer between sessions (somewhere between 5-15 minutes)

☝️ 1

That feeling when you need to go somewhere, but get drawn into an amazing talk! @arun.infy @useidel This is awesome!!! “Silos!!!”

👍 2

Thanks @genek101!


Great work!!!!

Chris Gallivan, FCA, Builder of JOY19:10:52

None of our devs have access to the cloud :(


I love these highly produced videos that teach us about companies, but especially so when they focus on the tech org!!!

Jess Meyer - IT Revolution (she/her)19:10:13

Continue the conversation with speakers at #ask-the-speaker-more!


Muse Dev is super handy. I hope everyone gets to check it out. Great to see an update on the survey. Good segue to dev journey talk.

❤️ 1
Matt Cobby (Director of Engineering, Deloitte)19:10:54

How do you metric your inner source ecosystem? @arun.infy


We measure the number of pull requests that come from developers outside the project (or who dont have access to the repository)

Matt Cobby (Director of Engineering, Deloitte)19:10:20

Is there tooling you have created for that? It's the automation of that internal vs external people that is subtle.


We use various dashboards (splunk/elk) to be able to define this. Our team structures are very well defined in active directory so it makes it slightly easier to generate this

Ian Silverwood (IT Manager at Ubisoft)19:10:39

@arun.infy @useidel How would you foster inner sourcing in an environment where teams are working on different products, using different tools, and are worldwide?

Udo Seidel19:10:02

We use communities for that, with webcasts, events, training and of course the same tools for collaboration and code management

Matt Cobby (Director of Engineering, Deloitte)19:10:18

World wide isn't a problem as such as the Inner Source as the techniques are asynchronous and generally use the written form

Udo Seidel19:10:06

Plus, we use also communities of communities ... decentralised

👍 1

To add to Udo - In general our teams are global, many teams have matrix reporting, this is in the culture of the company where teams have to contribute cross geographies. To aid innersource - we do market our products internally and make it exciting for others to contribute 🙂

Camilo Piedrahita - Bancolombia - IT Manager19:10:39

is it necessary serverless? we create dependency to cloud provider and its not cost effective in high volume application


Can you provide the context of this question so that I can answer it better :)

Ian Silverwood (IT Manager at Ubisoft)19:10:52

@arun.infy I believe Camilo means that if you integrate with AWS, you are slowly vendor-locking your software and then migrating to something else less expansive or another type when it's time poses a lot of challenges.

Ian Silverwood (IT Manager at Ubisoft)19:10:19

Feel free to correct me if it's wrong

Camilo Piedrahita - Bancolombia - IT Manager19:10:37

that's right @ian.silverwood we can use AWS, but we prefer to use EKS or another services...beacuse, if we are using only serverless. the vendor-locking is so complex and expensive

Ian Silverwood (IT Manager at Ubisoft)19:10:23

There are internal products on our side that have started to rise which aims to provide developers with wrappers toward vendor APIs/methods, to avoid this vendor-locking issue and enable easy switch. However, it has yet to be proven as the products are not live yet.


Its a tough question to answer 🙂 Will try to do my best, its a small subset which is using serverless on the cloud, they sure are vendor locked. But on the rest of the platform that we are building is completely cloud agnostic and easily portable across providers (public or private)

Andy Weldon19:10:00

@arun.infy Are your developers happy with using bitbucket rather than github?


@andyweldon Adoption of Bitbucket has been very successful. We were on legacy source code tools and migration to BB has been very swift. We also consume github for opensource contributions. The gitlab, bitbucket, github debate still rages on between the developers though 🙂

Rajat Bhatnagar19:10:50

@useidel @arun.narayanaswamy How does your SecOps team plug into your toolchain?

Udo Seidel19:10:00

Via several methods: first of all they are part of the wider engineering community, secondly we have developed a quite strong security mindset in the different teams, hence the SecOos people are seen as aliens and the are part of the tool selection/approval process

Rajat Bhatnagar19:10:47

Thanks for answering. So, given your scale, have you automated appsec with integration with your DevOps toolchains?

Udo Seidel19:10:55

As much as possible scaling out with white hats and similar activities

Rajat Bhatnagar19:10:12

Great! I will reach out with a DM to follow-up - thanks!

Zaren Balisi19:10:00

@arun.narayanaswamy I was impressed to see your daily builds on your Jenkins. May I ask if whether this is running on a public cloud utilizing k8 with their service, or did you set it up another way?


Its running both on private and public cloud. Its setup using images and containers both depending on the internal customer needs

Rajat Bhatnagar19:10:29

@useidel We're connected on LinkedIn (since 2015) - I will reach out to you on LinkedIn

👍 2

@arun.narayanaswamy You mention doing fun in your last slide. What all are you doing for fun to help foster the culture (a super important part of digital transformation vs companies that ad hoc rename teams as DevOps - cue the SMH Oprah moment - "You're now DevOps! And you, you're now DevOps! And you, and yoU!")? This is like hackathons, etc?

👍 1
❤️ 1

Hackathons is one, contests, ideathons, incentivizing automation, enabling gradual learning, enabling internal and external trainings, enabling sharing info with ease, providing opportunities to speak/attend conferences, t-shirts, stickers etc 🙂 @blakee do you have any ideas to make it more fun?

Jess Meyer - IT Revolution (she/her)20:10:27

Welcome our next speakers @claire.vo and @lawrence.bruhmuller!

Claire Vo20:10:51

👋 hey y'all

👋 1
Jeffrey Fredrick, Author-Agile Conversations20:10:15

I’m here because from the description I’m guessing there were some conversations involved…. 😄

Claire Vo20:10:32

Proof @lawrence.bruhmuller and I are real people. There was a bit of a hiccup with how our video files got complied! 😭

😋 1
☝️ 1
👍 2
simple_smile 2
😂 2

@lawrence.bruhmuller @claire.vo I love it! Feel free to send a new video to @jessicam and me, and we can replace it in the library. And thank you for this talk! Great interaction around chronic problems between Engineering vs. Product!

❤️ 1
👍 1
Jeffrey Fredrick, Author-Agile Conversations20:10:53

@claire.vo @lawrence.bruhmuller: how do you create that culture of trust?

Claire Vo20:10:24

It's one of our cultural values at Optimizely (it's the "T" in OPTIFY) so it's an overall company value

Claire Vo20:10:23

this is how we define that internally

❤️ 1
👍 1
Jeffrey Fredrick, Author-Agile Conversations20:10:45

That’s a fine definition of trust… however how do you get a culture of trust in practice? I ask because I know lots of companies that have Trust as a stated cultural value, but if you ask the people in the company they don’t feel a sense of trust across teams.

Lawrence Bruhmuller20:10:54

Yeah it is pretty core at Optimizely. But it's also just about leaders being willing to spend the time together, investing in a close partnership.

Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere20:10:10

@jtf my company is in early stages here, but we're experimenting with measuring psychological safety and Transformational Leadership via Google's Project Aristotle example and the ITRev "Transformational Leadership Quick Start" whitepaper as sources. That is at least helping us start to measure where we are at. Those are some of the behaviors that are trying to shift toward this mindset.

👍 2
Lawrence Bruhmuller20:10:38

Having each other's back. Showing vulnerability. Taking every feedback given with positive intentions and a "single team" mentality. Etc. 🙂

👍 3
Claire Vo20:10:29

No information or power hoarding. Admitting mistakes and celebrating failure. No drama triangles (make sure feedback is shared directly and privately.)

🙌 1
👍 3
Nick Eggleston, IT Architect (open to offers)20:10:35

@mring @jtf How did (does?) Google define and mesaure PS?


@nickeggleston See Has seven questions you can ask via a Likert Scale. We were able to experiment with this easily at a team level by re-creating via an MS Forms survey (for those who use Office 365).

Jeffrey Fredrick, Author-Agile Conversations20:10:54

@lawrence.bruhmuller @claire.vo: can you give an example of productive conflict you’ve shared? (love the phrase productive conflict btw)

Claire Vo20:10:21

The classic example (and I think we talk about this later) is the level of investment in tech debt. There's a real debate we have to face during planning and other times about how much engineering time do we put into things that are not customer facing. I think we both usually come into those conversations with a different point of view. We talk that out in the open. It's important for teams to see strong discussions with real debates.

Claire Vo20:10:39

But ultimately we end up coming together on something. Or know who is the decision maker and disagree and commit. But we have the discussion.

👍 1
Jeffrey Fredrick, Author-Agile Conversations20:10:26

Sounds like it would be a virtual cycle where the conversation to make the decision helps build the trust for future discussions.

👍 1
Lawrence Bruhmuller20:10:36

Another good example is around what amount of rigor and analysis is required for a given project ... balancing "small rocks" that teams can pump out quickly with bigger efforts that require a lot more thought, both product/customer related and also technically. Great push/pull discussions here.

👍 1

“Most important skill: Customer-centricity”. Nice.

Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere21:10:05

I feel like that one of five... something-somethings... :thinking_face: :unicorn_face: 😉

Brandon Brown - Nike - Sr. SW Engineer II20:10:16

My favorite from the business: "This isn't what we talked about."

Stephen Magill [Sonatype]21:10:04

how do you deal with this?

Pavan Kristipati20:10:57

@claire.vo and @lawrence.bruhmuller how do we overcome the initial thought that this arrangement requires very large IT teams? There would be lots of projects/efforts going on at once.. it would be interesting to know your thoughts..

Claire Vo20:10:18

I'm a big believer in fully staffed (meaning, able to execute independently) but relatively small teams. And limiting work in flight! We have a pretty big team at Optimizely but we also do alot (probably too much!)

Lawrence Bruhmuller21:10:56

@pavan.kristipati did we get to the question? Or are you saying that our approach only works once you have a larger team, and how would it work with a smaller team?

Andrew Hughes - Manager, DevOps Service Delivery QA (TRIMEDX)20:10:26

@lawrence.bruhmuller @claire.vo this discussion is amazing. I keep trying to type questions but you bring them up before I get them sent over 🙂

👍 1
thankyou 1
Andrew Hughes - Manager, DevOps Service Delivery QA (TRIMEDX)20:10:46

@lawrence.bruhmuller @claire.vo + DOES Scenius: Any ideas on key, measurable outcomes for re-platform?

Andrew Hughes - Manager, DevOps Service Delivery QA (TRIMEDX)21:10:41

I'm trying to advocate some measurable outcomes around things like Availability, Reliability, Performance, Security. Looking for a simple outcomes framework to overlay as a guide so we collectively talk about these things as we hyper-focus on feature parity with our legacy platform.

Lawrence Bruhmuller21:10:56

Depends on the goals. Developer velocity is one common one. Another would be more direct leverage for future projects ... "since we have this new platform / API / component this new project is considerably easier"

Randy Shoup (Chief Architect eBay; he/him)20:10:59

@lawrence.bruhmuller As we know, it's tough to make those major rearchitecture investments 🙂

Claire Vo20:10:19

I just lose an arm wrestling match to Lawrence that's how we decide to do them 😁

Lawrence Bruhmuller20:10:32

Haha. Good to "see" you Randy!

Lawrence Bruhmuller21:10:04

100%. Lots of the other things we talked about don't matter if we're trying to accomplish different things and our teams pull in different directions. One other benefit of an agile mindset is that this type of dissonance can't fester ... it's clear very quickly if you're misaligned and need to get back on track.

👍 1
Ben Dennerley - Trend Micro21:10:56

Each team owns their CI process, do they own the CI infrastructure too?

Peter Tiegs - Speaker - Intel21:10:09

Depends, We provide a common CI infrastructure that teams can use. Larger teams provided their own infrastructure

Eduardo Rodrigues Semensati (Procter and Gamble)21:10:02

what is the central CI toolset provided as an internal service by your team?

Peter Tiegs - Speaker - Intel21:10:15

Current central Set is GitLab (SCM) Jenkins (CI/CD), Artifactory for artifact managment

Eduardo Rodrigues Semensati (Procter and Gamble)21:10:37

nice. Any reasons why you are not using all the features in GitLab instead? Avoiding vendor lock down?

Peter Tiegs - Speaker - Intel21:10:55

Partially due to name recognition and what teams were comfortable with. That being said we are seeing an increase in GitLab CI/CD usage.

EmanuelMedina - Bancolombia21:10:08

HI!, How did you manage the mono repo against many teams (trunk based development ?) thinking in the streams of differents customers needs

Camilo Piedrahita - Bancolombia - IT Manager21:10:16

How can you guarantee government with so many toolchain?. for example reports for audit

Madhu Datla - Intel21:10:04

Compliance is key business requirement for Intel. We established standard security and compliance practices that every team is required to meet before the software is released out the door - we also have key infrastructure teams within IT that are responsible for security compliance of infrastructure

Eduardo Rodrigues Semensati (Procter and Gamble)21:10:23

interesting, but how can you standardize and drive internal knowledge sharing and innersourcing without standardizing at least some parts of the toolchain?

Madhu Datla - Intel21:10:32

We established source code mirror required for supporting triage and debug of issues - there are also branching guidelines to support different customer needs

👍 1
Scott Dedoes21:10:05

Are you using feature flags in your DevOps process? If so, are you using a system that you built yourself or a third party platform?

Peter Tiegs - Speaker - Intel21:10:20

We are not specifically using Feature Flags at this time

Scott Dedoes21:10:46

Is this because of the type of software that intel releases or it hasn't been a high enough priority to add into your workflow or some other reason?

Peter Tiegs - Speaker - Intel21:10:24

Most likely, much of the software that we release through this process is Drivers and Firmware, where it would be difficult or impossible to enable a feature after it has been deployed to a running system. If you have thoughts or recommendations on feature flags for this type of SW I would love to hear

Scott Dedoes21:10:39

That's good information and something I suspected which is why I was excited to hear this talk to learn more. I'm not an engineer, but I'm going to reach out to our team and discuss further and if I have a solution to help I will be in touch. My company, Split, has a feature delivery and experimentation platform developed by ex LinkedIn, Google and Salesforce engineers and our big focus is on the Experimentation piece to ensure you get value from the feature releases, but you are correct if you can't kill a feature then it limits the value but its an interesting topic to discuss. My ears really perked up when Madhu said "What cannot be measured cannot be improved" because that that is a big reason why we've been so successful. 😀

Scott Dedoes23:10:16

Hi @peter.g.tiegs I spoke with a few engineers on our team and as you mentioned unless there is connectivity to the internet to enable or disable features FF may not be possible to enable, but am assuming there are other business units that develop client facing SW or internal tools? You mentioned that Intel employs 15k+ engineers, are the majority releasing drivers and firmware or are there teams that release other types of software? Split has been used to help migrate services and testing of internal tools so that is an area where FF might be helpful, but most likely handled by another team.

Fred Ghahramani21:10:20

I did not hear you specifically talk about consistency in Architecture and Design fundamentals. Should I assume that is included in engineering piece?

Madhu Datla - Intel21:10:37

@fred.ghahramani Given the complexity of the system level software that each team delivers - the architecture and design guidelines are established and are localized to that specific module. The common interfaces are defined and agreed upon which lead to seamless integration

Peter Tiegs - Speaker - Intel21:10:47

@fred.ghahramani Yes we have not specifically tied any rules about Architecture and Design Fundamentals into out DevOps pipeline. One of the three teams that was mentioned (The Systems Engineering team) provides some coaching in that area. Generally it is up to the individual upstream IP SW teams

Fred Ghahramani23:10:11

thank you for clarification. Great story.

Eduardo Rodrigues Semensati (Procter and Gamble)21:10:57

3-5 years? With the speed of today and the amount of vendor-owned and open source tools popping up, isnt that too long? I need to setup a similar cadence on my end, but I am pending to have a 6 months evaluation cycle and 1 year adoption. Is it too small of a cycle?

😉 1
Peter Tiegs - Speaker - Intel21:10:17

3-5 years was a challenge for us communicate with our management. Their original asks were around 5 to 7 years. I agree accelerating this eval period would be great. It can some time take over a year to role out a new tool across the organizaition

Eduardo Rodrigues Semensati (Procter and Gamble)21:10:58

yes, I know where you come from. Our own Enterprise Architecture and Application Portfolio Management cycles in P&G work on a exact 3-5 years cadence (something we call the Domain Master Plan), similar to what you have. I do however believe that in this space we need to be more agile and be ready for changes here and there to keep up with innovation and avoid vendor lockdown

Peter Tiegs - Speaker - Intel21:10:12

@emgomez Our mono repo is repository that acts as an index to various other repositories. A manifest file in that repo has a unique set of references to the various IP SW teams at each revision

👍 1
Scott Dedoes21:10:02

Thanks for a very informative talk and for taking the time to answer our questions!

Eduardo Rodrigues Semensati (Procter and Gamble)21:10:55

thanks for the session! One question: is that session supposed to be shorter? I see that the next item in the agenda come only at 45 past the hour, in 20 min from now

Michal Jackowski Procter and Gamble21:10:19

Hey Eduardo! I have the same question and I'm concluding that presenters just finished ahead of time.

Rob Parkhill, Director SW Engineering, Hexagon AP21:10:11

The video in the library is 27 minutes long, so they just finished ahead of schedule it seems.

Peter Tiegs - Speaker - Intel21:10:26

Yes we had a shorter presentation

👍 1
Michal Jackowski Procter and Gamble21:10:43

Still good one! thanks a lot!!

Rob Parkhill, Director SW Engineering, Hexagon AP21:10:24

How are you allowing the "low level" software teams (device drivers, etc.) to do CI on in-development hardware?

Madhu Datla - Intel21:10:19

great question @rob.parkhill524 - the hardware goes through long process of development and is extremely distrubuted in nature (across the globe) - we have a combination of localized validation where each team does CI on their own hardware and a centralized CI system that we showed in third foil which is doing a global CI

Rob Parkhill, Director SW Engineering, Hexagon AP21:10:29

So for the long hardware dev process - are you following a more waterfall method (strict requirements, solid systems engineering principles, etc.) up until the hardware is "good enough" for the teams to move into a more agile/CI process? Or have you figured out how to do hardware dev in an agile way?

Peter Tiegs - Speaker - Intel21:10:51

Adding to what Madhu said, We deploy early HW and SW Simulations of HW to out SW teams for use as part of their CI pipelines. One area we are exploring is creating a hybrid cloud of this HW

Rob Parkhill, Director SW Engineering, Hexagon AP21:10:47

I ask because we are struggling with this right now - when developing new hardware, we follow a rigid waterfall model even for the software component, but then new SW capabilities after that initial development are much more agile.

Rob Parkhill, Director SW Engineering, Hexagon AP21:10:19

So simulators/emulators for in-development hardware?

Peter Tiegs - Speaker - Intel21:10:50

I would say the the HW is still developed largely in the Waterfall model, But we have started to reach across the aisle to our HW design colleagues to see how we can apply CI to what they do.

👍 2
Peter Tiegs - Speaker - Intel21:10:26

And yes Simulators and Emulators when we can.

Madhu Datla - Intel21:10:31

Yes. as you are experiencing hardware dependency will lead to longer development cycles - that is why we have pivoted to use models/ simulators early on so the software development is not lagging till the HW is fully ready.

👍 1