DER CONFEXX CONSULTING EXPERTEN BLOG
Talend Cloud – Battle of the Engines
Cloud vs. Remote
To be honest, it's not really a fight. After all they both play in the same team and can be used hand in hand. But what are the differences between both variants and when to use what engine?
To be really able to judge you need to understand what makes a Cloud Engine and what makes a Remote Engine.
Cloud Engine - Plug & Play
As a proud owner of a Talend Cloud subscription you are provided at least one Cloud Engine or more, depending on your contract. Your Cloud Engines will be completely preconfigured so you won't have to overcome any technical obstacles. You can start right away. Talend charges you with so called tokens. Providing a Cloud Engine costs you 45000 tokens. Cloud Engines are hosted in Amazon Web Services (AWS). It is planned to have your Cloud Engines in Microsoft Azure as well by the end of this year (2019). There are no other costs for you next to the 45000 tokens. It seems obvious that due to this fact there must be some restrictions when using the cloud engines. This is simply due to the fact that Talend won't be able to provide unlimited computing power after paying this one time fee.
As the name indicates, the cloud engine runs in the Talend Cloud. Its data is being exchanged between the cloud engine and the connected cloud SaaS services. Communication and orchestration is directly handled in the cloud between the TMC (Talend Management Console) and the cloud engine.
Cloud Engine facts
|(+) comes preconfigured||Maximum memory of 525 GB|
|(+) AWS Cloud (Azure to be released soon)||Temporary space of 200 GB|
|(+) for Data Integration||2 VCPU's|
|(+) Maximum of 3 jobs in parallel||Maximum of 225 GB per |
|(-) Talend API support not yet |
|Maximum of 3 jobs in |
|(-) Pipeline Designer support not yet |
|5000 tokens per engine|
Remote Engine - the Allrounder
The small brother or more likely the taller one? "Small" if you take into consideration that you "only" need 9000 tokens for a remote engine. "Tall" if you think of what the remote engine is capable of. What the cloud engine is not able to cover is easily executed by the remote engine. Being sold often as a hybrid-approach in order to serve the needs of the local requirements of the customers and to simplify the transition into the cloud, it is becoming an allrounder. With being available in the AWS as well as in the Azure marketplaces, the remote engine can be used in nearly every setting, may it be on-premise or in the public cloud. The usage of a remote engine is then only limited by the available compute power. The remote engine is available for Windows as well as for Linux run systems. Other costs only occur for the provided infrastructure, whether it be the customer's data center or - in case of a public cloud - the provided virtual machine. The remote engine can be operated from behind a firewall. Its data is only being exchanged between remote engine and its connected services. Communication and orchestration is carried out over a secured https connection to the Talend Cloud, wherewith your data is secure.
Remote Engine facts
|(-) needs to be installed and configured||Only limited by the server's capacity|
|(+) platform independent||9000 tokens per engine|
|(+) Unlimited amount of parallel jobs|
|(+) Talend API Support|
|(+) Pipeline Designer Support|
|(+) Available on AWS and Azure marketplace|
Cloud, Remote or Hybrid - a choice that will help you
How does the customer benefit from having the choice and what are Talend's hopes in offering the two engines? It sounds very much like a rivalry between these two engines. Why should I use the Cloud Engine if the remote engine can do most of it even better? Simply because: For one, Talend creates the highest level of flexibility for any use case and in any phase of a company's transformation. For two, Talend covers a broad spectrum of customer requirements, no matter if you are already in a cloud, you are just starting or you are planning to move to the cloud at a later point in time.
Different benefits for different situations
You don't use any cloud services by now but like to develop into a serverless direction?
With a hybrid solution you can both access your existing infrastructure via the Remote Engine as well as taking first steps towards a serverless architecture with the Cloud Engine.
You already use one of the big players in the cloud business?
With the help of a cloud engine you can start right away with implementing your requirements in the cloud. Nevertheless, a remote engine can also be a viable option due to being scalable when you need it.
You use cloud and on-premise services?
Also in that case a hybrid approach in the Talend Cloud would be the right option with offering sufficient security for your (local) data. At the same time it offers a seamless transition into a serverless architecture, as soon as older infrastructures are about to be replaced. With Cloud Engines or Remote Engines in a Public Cloud also all other imaginable scenarios are covered.
You are currently happy with your on-premise solution but might want to change later?
Here the Remote Engine can be recommended. Your services will still run locally and no customer data will be exchanged between the Talend Cloud and the Remote Engine. Only the communication and orchestration, e.g. logging or status messages, getting transfered through a secure HTTPS connection.
You move data between Cloud Services?
In this case the Cloud Engine is a simple solution. If you have to deal with more complex requirements then the Remote Engine in the Public Cloud will fullfill your needs.
You would like to combine Cloud Services with On-Premise Services?
Talend Cloud can also cover complex use cases between Remote and Cloud Engines. You might want to create procedures that automatically fetch data from a SaaS Service via the Cloud Engine. In a second step this data could be processed in a Remote Engine to, for example, combine it with Local Services, save it in a Data Warehouse and provide it again with an API.
An easy way to the future
Cloud and Remote Engine - both got its benefits and added values and they are optionally combinable. Also are they easily configured with a Talend Cloud subscription and ready to implement in nearly every infrastructure. This leads to a maximum flexibility in meeting todays and future challenges for your company.Weiterlesen
Managing your SSH-keys
Giving your users access to systems can be a very challenging task. Especially, when the team is growing and changing over time it can be quite a challenge to keep an eye on every login. It is also considerably insecure to use passwords since they can be leaked easily even without intention.
A convenient and secure approach to these challenges is to completely shift all authorization processes to SSH (if possible). Some processes may need SSH2 (like Talend), and some need special algorithms to work (Talend only works with RSA). You can use PuTTygen or Talend itself to create the keys. The hoster (Azure DevOps) requires the public-key in OpenSSH-format.
In this post we will manage our SSH-key for Talend to connect it with Azure DevOps.
Generate Key via OpenSSH
The keys are managed via the "config" file inside your ~/.ssh/ directory. If you have OpenSSH installed on your system you can use this script:
# SSHPATH: dir where key will be stored. # If dir doesn't exist, it will be created SSHPATH="~/.ssh/" # NAME: Name of your key. # It is adviced to name it after used algorithm and/or usage. NAME="id_rsa" # TYPE: Name of used algorithm. # rsa is default and commonly used, while ed25519 is recommended, # but may not work with everything TYPE="rsa" # SIZE: Size of used key. # The bigger the better, but slower in generating. SIZE=4096 ssh-keygen -f "$SSHPATH$NAME" -t $TYPE -b $SIZE -C "$NAME"
Generate Key via PuTTYgen
Alternatively, you can use PuTTYgen in case you prefer a GUI. It is recommended if your OpenSSH doesn't have SSH2 yet.
Make sure you select a SSH-2 RSA key:
Now click on generate. To create randomness for the SSH-key you will have to move your mouse around. After the process is finished you should see this:
For easier management of the keys it is adviced to add a key-comment within your E-Mail.
Save your private key in your home directory in the .ssh folder. To do so just click on Conversion > Export SSH-Key. Name it id_rsa.
Now copy the contents of the new box or save the public key into a file (via Ctrl + C):
Putting your key on Azure
To get your new secure key on Azure you will have to login to your DevOps account. https://dev.azure.com
On the top right corner please click on your profile and go to security.
Go to your "SSH public keys" and then click on Add.
Paste the public key that you generated before in the provided field and name it Talend.
Conduct your keys
Add this lines to your config-file (~/.ssh/config) :
# Talend Host vs-ssh.visualstudio.com Hostname vs-ssh.visualstudio.com Port 22 User *YOURUSERNAME* IdentityFile ~/.ssh/id_rsa
NOTE: In case your firewall blocks this port, you can also tryout other ports (e.g. 443) with this method.
You can also provide SSH-keys for each project seperately. Simply put the repository-link into the "Host" and keep the Hostname. This ensures full control for both client and server for each login, and simplifies management for admins.
To test your key you can try this command in your SSH-supported console/bash:
ssh -vT vs-ssh.visualstudio.com
And voila, you are done!Weiterlesen
Moving Your Use Cases Slowly Towards Talend Cloud – Step by Step
In most organizational scenarios you will not be rolling out your use cases to Talend Cloud with a big bang – meaning that you turn off the old world and turn on the new world.
Getting There - Step by Step
In most cases moving to the cloud will be a step by step process. You will move some of your use cases to the cloud, run and test them there, experience some differences, some limitations maybe and learn gradually what the best practise approach is. This is an iterative process.
In order to prepare yourself, you can setup a trial cloud instance as a playground for your first steps with the Talend Cloud. In addition, you can do some online courses offered by Talend, e.g. Cloud Essentials. This really helps you to get familiar with the enviroment.
We can help you every step of the way to your Talend Cloud. We can have workshops with your tech guys, we can set up your infrastructure to work with Talend Cloud - whatever you need in order to successfully migrate to Talend Cloud. In your own pace and tailored to your needs.