ActiveMQ Training Shorts - Messaging Background
Matt Pavlovich discusses the role of messaging and ActiveMQ in addressing challenges in distributed computing. He explains key terms like broker, producer, consumer, and subscription, and categorizes outages into planned and unplanned. He highlights three key fallacies of distributed computing: unreliable networks, finite bandwidth, and non-zero latency, and how ActiveMQ helps mitigate these. Pavlovich emphasizes the importance of standardization for enterprise applications, especially with the rise of microservices and multi-cloud strategies. He notes that brokers provide asynchronous operation, bandwidth mediation, and decoupling, making applications more flexible and efficient.
Video Contents
I want to talk about just a quick background about messaging and just to just to level set, what are some of the problems that solves in distributed computing? So we're often asked, When should we use messaging ActiveMQ, or event driven architecture? So the purpose of this segment is to understand the distributed computing problems that ActiveMQ addresses for your application teams.
From a terminology perspective, we use these terms. We have a broker. This is the server side process. This is essentially the ActiveMQ server or the ActiveMQ broker. So it's the process that receives messages and manages distributing them to applications, then we have a producer. A producer is the application side component that's sending messages. The consumer is the application side components that is receiving the messages. A subscription is messages stored for a specific consumer. And then when we're talking high availability and clustering, we like to break the outage terms into a planned outage or an unplanned outage. So in the case of the server side outage, these describe when a broker is no longer available to serve producers or consumers. So example of planned outages would be things like upgrades, patching, scale downs or a decommission an unplanned outage would be something like a resource exhaustion, a bug or a crash. It's helpful to separate all the types of outages just into these, these two categories, because then we can plan how we handle those, how we manage and monitor those things, instead of trying to write playbooks for every type of scenario. We can just categorize them into the two a planned outage or non planned outage.
So ActiveMQ, messaging and event driven really solve a lot of the challenges when working with distributed computing. So when you're working with data or invoking processes on a single computer. Everything is very straightforward. It evolves and involves a relatively few number of problem scenarios. So if you think about writing an application, if you call another method, or you want to pass a data structure to another service in the same runtime, or another process on the same machine, that's very reliable. It's very there's a lot less moving parts. There's a lot of less layers involved in patching to make sure that that's going to occur. Once we start moving data between computers, things can get tricky. Same thing with invoking processes when we're running those remote process and remote computer can become challenging. So there's a number of key fallacies.
So there's a nice reference to the key to fallacies of distributed computing. There's eight of them. We want to focus on these three as it relates to event driven and process and vocation on remote systems. And we rewrite these. We flip the logic on them to make them more straightforward. So essentially, the network is never 100% reliable. This means the path that our two computers are talking to also needs maintenance. It needs updating, it needs security patches. It needs to be reconfigured. This is especially the case in the cloud when we've got Kubernetes and nodes and patching and all these things going on all the time that the network layer also needs to be able to handle change, and so the applications on top of that need to be able to handle these types of blips and outages and reconnects. Also, the network bandwidth is never infinite. So the amount of data that we can pass between two computers is fixed and also may vary depending on, you know the situation and how what other processes are using that same shared network. Also, network latency is never zero. Latency is one of the trickier things to tackle. Latency is essentially the time it takes, you know, to round trip a piece of information or a message, while bandwidth may be massive, sometimes latency can be higher, resulting in lots of slow weights, and so optimizing for latency can be a real challenge, and one of those things that ActiveMQ helps us overcome.
Application developers, one of the things that we want to keep in mind is application developers have a need to solve challenges. They have a need to move data between applications. They have a need to invoke processes on other remote systems. However, we want to be sure that they're not reinventing the wheel every time, so they don't hit these same challenges, those same policies, or these same issues when working with data on remote systems or invoking processes. So having every application account for all these distributive problems is very inefficient. It also becomes challenging for administrators and application support teams that they need standardized approaches to solving these problems so you can consistently support applications in production. Most Fortune, 500 companies have 10,000, 20,000, 30,000 applications running and. And the more each of those applications do things in a different way, the more challenging it is to support and maintain all those applications. With the rise of micro services and event driven and smaller deployments, these numbers are going to quickly approach the millions of deployments and modern enterprises. So it's super critical that we work to educate developers to focus on standardizing how they're doing remote communication and remote data transfer.
On the flip side, we also have when we talk about managing, so we have the actual function of moving data, invoking processes, then we have the managing the solution at scale. And so these would be enterprise technical debt pieces that can cause complexity and ensuring that the enterprise can stay nimble and agile and flexible to be able to react to the changing needs of the business. If we build our order to quote system right to be tightly coupled between each other. That makes it very difficult to add another data warehouse or another AI insight engine to grab that information in between. So we want to limit tightly coupling components. This gives us more flexibility to use data without having to update every application that's involved all the time, we want to be able to have applications operate on their own, and other applications come around and essentially get into those data streams right when they need to, without having to involve the other applications. Another challenge in enterprise technical that is a lack of alignment to standards. So this is really key. This is something that we feel like is also coming, going to become a bigger highlight here as we look into Cloud standardization, migration and multi cloud strategies. If we're building multi cloud strategy, it's really critical that our application teams are coding to some sort of standard that can move across cloud providers, whether it be a private cloud or out in the public cloud, because if we're tying to a cloud-vendor specific technology or cloud-vendor specific API or protocol, that's going to limit the ability for that component to move, migrate or be involved in a multi cloud strategy. And there's also a lack of knowledge. This one gets tricky when one of the things, a lot of times it's, it's as developers or administrators get into the technology, you know, the first 80% of the information could come really quickly. But then there's, there can be sharp corners or edge cases that become a challenge. And so the more we standardize the deployment, the more we standardize the application side, the more we can harness this knowledge right and componentize the knowledge into standard cluster configs, standard coding patterns. We're not requiring everybody to become an expert. We're not requiring everybody to become a committer on an open source project, just to use the technology to implement a business need. So the goal here would be to enable application teams to quickly develop and address the needs of the business, and on the operations side, provide highly reliable, fast, efficient, cost-effective messaging solutions as a service.
So this is how messaging solves it. Brokers operate asynchronously, so that helps solve the network reliability. So in applications or network is are offline, the brokers essentially hold the messages or queue the messages until the network comes online or the other application comes back. Brokers mediate bandwidth between producers and consumers. So if a producer is faster than the consumer, the broker helps prevent the consumers from being overrun or flooded. Brokers allow producers and consumers to operate independently. So if one of the other has higher latency, if the producer component, for example, is sending over an overseas connection, but the consumer is local to the broker. You're not tying up that consumer for that high latency, which involves keeping threads around, requiring the consumer to have a higher contention. So the broker helps mediate that. The components are not tightly coupled. So when producers or consumers are not directly connected, this allows other applications to come along and get access to that data stream without having to recode the producers or consumers. This is a really critical Enterprise Architecture key flexibility point that differs from APIs.
The broker implements standard protocols and APIs, so when applications are coding to JMS and ActiveMQ, they can implement this on-prem on physical metal, in private cloud and public cloud and any of the public cloud providers. So you have tons of operational flexibility as well as you know, business operational capability to pick the best cloud solution for every application needs to be deployed. And lack of knowledge, brokers isolate a lot of the hard stuff when it comes to being able to implement a lot of these things, thread management, writing data to disks, keeping track of acknowledgements, keeping messages in order. There's a lot of things that the brokers isolate that developers and administrators don't have to, you know, become experts, to become committers on, you know, be effective at implementing these solutions. In the last piece here, really just about how messaging compares to APIs. It really boils down to one, one big difference. Messaging is both design time and run time, decoupled between endpoints. And that last part, the runtime decoupling is the most critical. APIs are design time decoupled, but at runtime, they're tightly coupled. So APIs suffer from various fallacies and technical debt. However, you know here at HYTE, we know that there's value in using both. And so if you apply the CQRS, there's an easy way to ensure you've got strong Enterprise Architecture capabilities that if you use essentially APIs for queries or reads, then you eliminate the majority of the fallacies, and you only take on some minor, minor coupling, but the key piece there is make sure you're using messaging for commands, anything that's sending us a data change or invoking a remote process.
