Our CTO, Tony Rost, sat down with engineering manager Nathan Wilkerson to talk about his key takeaways from this year’s re:Invent: Chaos Engineering...
Agile Media Workflows + Cloud
Executive Interview with Lincoln Spiteri, VP of Engineering at Dalet. Lincoln has been in the telecommunications, media, and software industry for over 20 years. He is currently the VP of Engineering at Dalet where he drives engineering excellence on solutions that help customers manage, curate, orchestrate, deliver and monetize their content.
This blog is one of a series of Media & Entertainment Cloud Ecosystem interviews.
Executive Interview with Lincoln Spiteri, VP of Engineering at Dalet
While in-person events have been curtailed by COVID-19, innovation continues to advance in both tech and media and entertainment. I’ve been sitting down (virtually) with leaders of some of the leading AWS Partner Network (APN) Technology Partners involved in the M&E ecosystem—and their insights can offer a lens into the future of the industry.
Lincoln Spiteri has been in the telecommunications, media, and software industry for over 20 years. He is currently the VP of Engineering at Dalet where he drives engineering excellence on solutions that help customers manage, curate, orchestrate, deliver and monetize their content.
In this interview, Lincoln and I chat about end-to-end media management, the importance of metadata, and what he sees for the future of the industry.
Q: Tell us broadly, what does Dalet do and how does it fit in the media and entertainment space?
A: Dalet is a technology company building solutions and offering services across the breadth of media and entertainment. We facilitate the creation of content—particularly in newsrooms, for example—and management of content. So, managing large archives, large libraries of assets, high-grade transcoding services, and also live ingest services.
The company’s been around for 30 years. We’re headquartered in France, but we have 19 offices globally, with 450 employees in total. It’s been a great journey. The company has grown through a number of acquisitions. The last one, which happened last year, was the Ooyala Flex Media Platform, which has been on its own journey for the last 11-12 years or so.
Q: Tell me about the product that you particularly focus on, the Flex product.
A: Flex started off as a workflow orchestration platform. The founders who conceived the idea behind Flex were working at the forefront of the transformation of media companies, going from tape to digital—fully file-based workloads.
The realization was this: as soon as you’re in that realm, the integration dynamics change, and now you’re talking about integrating IT systems rather than wiring up individual pieces of hardware. As we went along, we realized that doing the orchestration is valuable, but you actually need to have a handle on the assets that are under management. And if you’re managing assets and moving them around, the value is further increased if you’re also managing the metadata that comes along with the assets.
And then we found ourselves being asked, not to just handle media assets, but to handle any other type of assets—and also information relevant to the creation of assets. We can actually model different entities within the organization which can be associated with media assets as well.
So it became a more comprehensive platform—fully end-to-end—but our roots are media and workflow orchestration.
Q: That’s been quite the journey. You and I were both working on hardware-based solutions for video distribution when the rise of cloud began. First it was an infrastructure promise, where we could have fewer machines to wire in and debug. And then it was an infrastructure solution, a pricing solution. But then it introduced so many new opportunities around big data, around archiving—but most importantly, around delivering content to multiple screens through multiple mediums.
Now it’s the introduction of machine learning. And one of the things that I think is really fascinating is the rise of metadata, and how metadata has become the key player in orchestration. I’ve really enjoyed the journey of the rise of the cloud and how it has transformed management and distribution of assets over the past 20 years.
What has been your experience coming from hardware—from hardware custom solutions in the 90s and 2000s—to today? How did you get more involved with the cloud personally? How did you evolve and start advocating for cloud solutions? What was that experience like for you?
A: My first experience in media was running a head end for digital terrestrial television. It was your classic hardware racked-up: individual encoders, hardware multiplexors, receiving feeds from satellites and mixing them and redistributing them. So it was a completely different proposition.
The way we used to receive content there —and perhaps this is still being done— was coordinating download from satellite feeds and literally recording that and playing it back out. Those days are far gone; that sort of contribution nowadays is done only in very particular situations.
My next step was to move away from media and work for a company that asked us to build our own infrastructure as a service. We built a data center back in 2008, and part of the vision was to offer cloud services. This was at the very beginning when AWS was still emerging. Netflix was having success in AWS, even though they had two wobbles in quick succession that showed the pitfalls of cloud, early on. So I have experience building the machinery of cloud infrastructure: how you create and deploy VMs and the networking fabric that pulls everything together.
Obviously, things have moved a lot since then. I’m talking about 2009-2010. But when I joined Nativ, which is where the Flex Media Platform started, it became clear that the transformation from tape to file-based systems does lend itself to the cloud. So, very early on, even though our customers were saying cloud is a long-term goal, they were also looking at it.
We were doing some work here in the UK with ITV, for example. They were going to modernize, but they were still running most of their workloads on-prem. And it wasn’t an easy task, especially when you don’t have direct control over the hardware. But there were some experiments we’d done, and some experience we had gained—our founders had worked with the BBC, and the BBC was already utilizing AWS services with everything. They were early adopters.
So for us at that point, it was obvious that the next thing was going to be digital. Rather than having big rooms filled with tapes, now we would have big rooms filled with racks, and those racks are expensive, and they require a lot of maintenance. They break, and you have to do disaster recovery, and it becomes extremely complicated. Media companies want to create media, and I think the cloud providers are such a comfortable proposition. You could see it coming from miles away.
So, in 2014, we started the transformation from monolithic software to microservice architecture; making the cloud native and starting a long transformation of our software product that would hopefully fit, hand in glove, with the future that we saw coming. And so far, so good. I think we’ve predicted well. Obviously, there’s been some accelerations along the way. I think it’s an evolution, and a pretty organic one so far.
Q: Yeah. Let’s talk about today. With the portfolio of solutions you have, where does the solution set fit in the AWS media ecosystem? How does it complement AWS Elemental, MediaLive, or MediaStore? If I’m currently using MediaLive or MediaStore or Elastic Transcoder, for example, how does the Dalet solution set complement or work in that environment?
Q: Part of what we try to achieve with Flex is an integration backbone that allows you to access services like those in a uniform manner—to then be able to use them in, say, workflow automation. For us, anything that has an API is game. We’re happy to integrate with that. So, we have integration into the Elemental transcode services and MediaConvert. It’s a fairly natural fit.
And it’s not just about starting a transcode—there’s all the automation that can come with a system that is orchestrating and coordinating the work, such as scheduling the start of a recording. You’re putting more information in there.
That’s what we’re trying to achieve with Flex. It’s hiding the complexity of standing up the systems and using them—using them effectively and cheaply. Because ultimately, cloud is a blessing, but it’s also one that can be an expensive proposition. We’re trying to enable our customers to use and consume cloud resources in a responsible way that’s not going to break the bank for them.
Q: One challenge I’ve noticed a lot of customers have is that the cloud is evolving faster than their internal capabilities can keep up with. I spend a great deal of time being a guide and advisor on the latest and greatest in cloud technologies. It sounds like the solutions that Dalet has are doing the same thing. They’re trying to enable customers to adopt the best practices and standards of the cloud, to take advantage of that, without causing them a headache.
Metadata is something that I thought was a second tier topic for a long time, but now it has become a first tier topic. And largely, it’s because metadata is not just serving EPGs, feeding codes, that kind of thing. It’s the backbone that orchestrates delivery across multiple screens and multiple mediums.
What are the trends in the importance of metadata and metadata management that you’ve noticed, and where do you think the industry is going with standards around new types of metadata schemas? And maybe even machine learning augmentation of metadata?
A: Well, we are no longer dealing with videotape archives and pieces of paper; everything is digitized. And that creates a problem, because suddenly you have the ability to store as much data as you want. It becomes an infinite space.
We are seeing a lot of our customers wanting to store as much data as possible—both descriptive metadata and temporal metadata describing what’s happening within a piece of content. And also, as I described earlier, associating business metadata to a piece of content. So it’s not just content-derived data, but also stuff that drives the business, like budgeting.
In terms of standards, our mission is to embrace standards—but not necessarily be rigid and driven by standards. One of the things that I think is important—and what the cloud provides at a large scale provides—is agility.
Standards are good when you’re moving content between different providers. Standards have a big role to play there. But in your day-to-day business, you want your metadata schemas and those things to be malleable. They need to change. The business is always transforming itself. When you’re transforming at a fast pace, you’re going to get things wrong. You may want to revisit certain assumptions and either capture more data or adjust the structure of the metadata system.
But that definitely starts verging on a big data problem. Now you have all this data, which you are often capturing through automated means—maybe using AI augmentation, which we’re seeing more and more of. Dalet has products that allow you to aggregate different AI engines and then build the results, which we can then associate to the asset.
But then you have all this data—what do you do with it? How do you find it? Search becomes a big part of the value proposition. And having good tools to access the catalog, through all the various facets that the metadata is offering, is a very important aspect.
It’s a time of plenty. People do, because they can. And then they find they’ve created another problem. But luckily it’s one that computer science solved a long time ago. We are enabling solutions toward this new problem that we’ve created for ourselves. But they’re good problems to solve, and knowing more about your data—especially for archivists dealing with historical data—is important.
One thing that we can’t really remove from this equation is the humans. They are ultimately the best form of intelligence. AI is great; it’s not perfect. And there’s always human stuff that a fully automated AI system—will not be able to achieve.
Building the tools, having an integrated tool set around metadata and search—which allows human operators to actually work efficiently and effectively in managing assets and the metadata that comes with those assets (like having strong vocabularies and a consistent metadata set)—is very important. And that ultimately is what’s going to allow you to transform your metadata to any standard, at the interface between interchanges.
So there’s things like IMF that we’re seeing more of, and having well-defined schemas at that level are important. But once you’ve acquired the content, you can’t limit yourself to whatever that standard is offering. You need more. And you need it in the shape you want it to be, rather than what the standard authorities are telling you it should be.
Q: Yes, one challenge that I cannot figure out how we’re going to solve is how the origin of metadata—which is often done in a production house or by someone writing a press release—how the origin of metadata, the first person to write it down, can play nice with downstream optimization and downstream standardization. It’s been a recurring challenge that I’ve been exposed to.
We can build great systems all day long, but the actual first person to write down the data in the first place is usually using some other technology or just email or spreadsheets, and that really cuts out a big challenge for us downstream. We’re trying to turn that into something that we can take action on. And hopefully we’ll find a solution for that too.
A: I think that’s the beauty of cloud platforms and the movement of systems onto the cloud. Predominantly, they become a web-based system and everything is driven through the browser—and that makes it a lot more accessible. You can make specialized tools available remotely and package them in such a way that you don’t need to be a super-duper expert operator to do things. Ultimately, you’re putting things in the browser, and people expect a fully browser-like experience: simple but powerful and effective. And you get those tools to those people—I think that will help clean things up a little bit.
Q: Let’s hope so. AWS has two broad tiers of machine learning services: the abstracted application tier—like Rekognition and Polly and Comprehend—but they’ve also got the Sagemaker tier too, where you can build out your own ML architectures.
Where is Dalet exploring and creating solutions today? Are they building deep learning algorithms on their own architectures? Are you incorporating Rekognition and Comprehend and those types of solutions?
A: We have Dalet Media Cortex, which runs in AWS for the serverless architecture—and from a technology point of view it’s leveraging that side. I’m not quite knowledgeable of the details of what they’re doing underneath there, but they are using those facilities and there’s some other algorithms that we are working on.
But beyond the analysis of media—video and audio content itself—I think there’s a case for AI. What I’m interested in, as far as Flex is concerned, is how do you make the factory so to speak that’s a bit more intelligent? If you’re dealing with a large volume of workloads and a large volume of file movements, how do you optimize that movement? What can you do to make the machine work more efficiently and make things happen at the right time, when they’re expected to?
Having the tools that AWS is offering definitely takes the burden off—the packaging they have around this technology makes it more accessible—to allow us to experiment and fail fast and learn quickly from what we’re doing.
Q: The mantra that I have is that the AWS stack with machine learning is really written for developers and has developers in mind. I’d like to convince even front-end developers that they too can experiment and develop solutions with that stack. It’s just simple endpoints with JSON. Hopefully, the development community really embraces that stack and incorporates it in their products.
So, we’ve talked about how we went from hardware to cloud and with that, the challenge to go from throughput to digital challenges—challenges that might be more complicated than some of the media and entertainment companies’ internal competencies can manage, and finally how products like yours help enable them and shepherd them to use the cloud appropriately. That’s today.
Where do you think we’re going with media management, media distribution, and cloud technologies in the next 3-5 years? What are some of your forecasts about the engineering and technology side in the media and entertainment industry?
A: I think more of the workflow will shift into the cloud. Amazon, for example, is making big moves—like the recent announcements around CDI shows their intent to have this portfolio of media-centric products. I think we’ll probably see this across all the different cloud providers, but definitely AWS has been at the forefront of this—and they continue to be.
There’s going to be more stuff moving into the cloud, but the consumption is going to change. And the relationship with the services is going to change. I think it will be more SaaS-based services, not necessarily managed by the content creators themselves, but perhaps through companies like ours or partners.
Also, services are going to be accessible to APIs, so media creators or content owners who have the capabilities are going to be able to utilize those APIs to their advantage—and do things that are not anticipated from the services that AWS, or even Dalet, is offering. Having that openness is certainly going to create some new use cases that will help media companies become more efficient.
We’re even seeing companies moving complete archives onto the cloud. When I say tape archives, this is digital tape archives, backup, storage—big media companies moving their “full show”, everything into Glacier. I think that’s going to happen more and more.
I think the next refresh cycles that companies go through are going to be looking to cloud-native, cloud-first solutions rather than, “Let’s build another data center, bring in more racks, and this and that.” I think those days are gone. I think eventually, we’ll get to a point with production where the bandwidth will be there to have multiple cameras streaming directly into the cloud, and everything will be happening in the cloud.
I think that’s where we’ll go. The capabilities are definitely there. Codecs are being prepared for those scenarios, the compute power is there, and there’ll be more of it as we go along.
Q: With this migration and move to cloud-first, I’m really looking forward to the opportunities for personalization in video, where I’m not stuck with a particular front door—my media experience can cross different brands, different streams. And a lot of that’s fed by metadata. But that requires, again, a really, really strong metadata universe and, in particular, an upstream metadata source that’s invested in that outcome.
My final question for you today: with the Flex product, what are you excited about for it in the next 12 months or so?
A: Well, we are currently working on making Flex ready for the future, which is SaaS. We already operate single-tenant SaaS in the cloud, which we manage on VPCs and our customers manage. But we now anticipate operating a multi-tenant SaaS business model.
That is exciting for us; it’s going to raise the bar of how we engineer things to solve the problems that come along with multi-tenancy. If your lights go out, then you’ve got a larger problem to deal with—so we’re working increasing our standing with regards to reliability and those sorts of things. We’ve been doing a lot of work on that—and also integrating more closely with the facilities that cloud systems are offering, so that we have a more seamless experience and leverage the full potential of the cloud that our clients, big and small, can take advantage of. That’s hopefully where we’ll be.
Then there’s things that we are always mindful of. Security becomes a big concern with the public cloud, which means making sure you’re a good citizen and you’re deploying software that is secure and you’re following the right practices. That is something that is constantly under consideration, because as you’re doing more and more, you gain visibility. It becomes an important topic for everyone involved to stay secure.
Q: Absolutely. It sounds like you’re having a lot of fun.
A: Yes, definitely. And with the recent events globally, with COVID and everything changing around it, the demand is there. Previously people were sitting on the fence on whether they would go to the cloud or not, and now people are being pushed in one direction and everyone’s falling off the fence. Everyone wants to avail themselves of the services that we offer. Currently, we’re doing that with larger scale customers, but ultimately the problems that we solve are equally important for different types of customers. That’s certainly true in the M&E space, but it’s true for anyone who creates content—and basically everyone is creating content nowadays.
Q: Right. Acceleration’s going to be key, and that’s the value prop a lot of folks are looking for: can you accelerate me into the cloud?
A: For sure.
Q: Thank you so much for joining us today. If anyone wants to get in contact with you and learn more about Dalet, where can they go?