indiTraders - Forum for the Active Indian Trader  

Go Back   indiTraders - Forum for the Active Indian Trader > Technical > Auto & Algo Trading

Reply
 
Bookmark and Share LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 08-07-2016, 03:04 PM
ajrj ajrj is offline
Regular
 
Join Date: Jul 2012
Posts: 56
Thanks: 190
Thanked 125 Times in 55 Posts
ajrj will become famous soon enoughajrj will become famous soon enough
Default Cloud servers for Algo Trading

Mod Edit: Moved posts on discussion regarding use of cloud servers for algo trading to this thread.
Originally Posted by NJ23 View Post
I think there might be some other issue because I used the same on an Amazon VPS with the internet speed of 300 Mbps about a month ago. I never faced latency or any freezes of the software. The customer support is not that great I agree to that.

How much are you paying for Amazon VPS ?

I read their TOS & got somewhat confused about the 750 hours free monthly limit. Even if we take 31 days & 24 hours then it comes to 744 hours in a month..

Whats the catch or terms & conditions in this , also I do not use Credit Card, so does that mean I cannot subscribe their services ?

Thanks.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
The Following User Says Thank You to ajrj For This Useful Post:
  #2 (permalink)  
Old 08-07-2016, 03:49 PM
NJ23's Avatar
NJ23 NJ23 is offline
Regular
 
Join Date: Jul 2014
Posts: 50
Thanks: 65
Thanked 83 Times in 43 Posts
NJ23 will become famous soon enoughNJ23 will become famous soon enough
Default

Originally Posted by ajrj View Post
How much are you paying for Amazon VPS ?

I read their TOS & got somewhat confused about the 750 hours free monthly limit. Even if we take 31 days & 24 hours then it comes to 744 hours in a month..

Whats the catch or terms & conditions in this , also I do not use Credit Card, so does that mean I cannot subscribe their services ?

Thanks.

Yes, that's just for the first year. The 750 hours free for a year. After that, you'd have to pay regular prices for your configuration. You can find that out on their pricing calculator. I have no idea about if you can use any other payment method than a credit card. You can talk to Amazon. Even Google offers VPS named Google compute engine.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
The Following 2 Users Say Thank You to NJ23 For This Useful Post:
  #3 (permalink)  
Old 08-07-2016, 04:08 PM
no1lives4ever no1lives4ever is offline
Newbie
 
Join Date: Jul 2009
Posts: 3,295
Thanks: 19,010
Thanked 5,549 Times in 2,358 Posts
no1lives4ever has a reputation beyond repute
no1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond repute
Default

Originally Posted by ajrj View Post
How much are you paying for Amazon VPS ?

I read their TOS & got somewhat confused about the 750 hours free monthly limit. Even if we take 31 days & 24 hours then it comes to 744 hours in a month..

Whats the catch or terms & conditions in this , also I do not use Credit Card, so does that mean I cannot subscribe their services ?

Thanks.

You run an EC2 compute instance on a per hour basis. For 750 hours, you can run either 750 instances for 1 hour or run 1 instance for 750 hours.

Mostly if you run windows, the free EC2 instance is only enough to barely let you run the os and some apps.

You also need to understand how the amazon EC2 architecture works and how it is designed. At a very basic level your EC2 instances can be killed and restarted by amazon at any time. Any application that is run on Amazon EC2 needs to implement clustering to handle this basic situation.

Amazon AWS's EC2 compute instances are not the same as a vps provider. Used properly, it is way more powerful than any other hosting solution. But for most parts, if you are not making simple database backed websites where you are using amazon rds for databases and amazon's elb for load balancing, you will need to spend considerable effort in understanding amazon's architecture and adapting your application for the same.

Dont expect ec2 to behave like your regular shared or dedicated servers, even though they may appear to behave similarly. I recently had a client using another cloud provider suffer major loss of critical data because they deployed on a cloud platform without properly understanding how it works.

As a general rule, unless you are comfortable setting up a clustered environment for your hosting, dont host with amazon aws w/o getting your hosting architecture reviewed by a third party.

I would also recommend against using amazon aws for hosting windows based auto trading apps unless you have handled the scenario where your host can be restarted randomly by amazon.

The same applies to google's compute engine and other cloud based virtual servers. These are not like your traditional VPS.

-- no1lives4ever
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
The Following 2 Users Say Thank You to no1lives4ever For This Useful Post:
  #4 (permalink)  
Old 08-07-2016, 05:55 PM
NJ23's Avatar
NJ23 NJ23 is offline
Regular
 
Join Date: Jul 2014
Posts: 50
Thanks: 65
Thanked 83 Times in 43 Posts
NJ23 will become famous soon enoughNJ23 will become famous soon enough
Default

Thanks no1lives4ever.

Can you elaborate why clustering is not a good idea for a trading virtual server? How does it affect anything? Cause I have not encountered my machine restarting, as in the way we start our laptops and desktops, since I've started using it.

Big whoop! Not a tech person.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
The Following User Says Thank You to NJ23 For This Useful Post:
  #5 (permalink)  
Old 08-07-2016, 09:52 PM
ajrj ajrj is offline
Regular
 
Join Date: Jul 2012
Posts: 56
Thanks: 190
Thanked 125 Times in 55 Posts
ajrj will become famous soon enoughajrj will become famous soon enough
Default

Originally Posted by no1lives4ever View Post
You run an EC2 compute instance on a per hour basis. For 750 hours, you can run either 750 instances for 1 hour or run 1 instance for 750 hours.

Mostly if you run windows, the free EC2 instance is only enough to barely let you run the os and some apps.

You also need to understand how the amazon EC2 architecture works and how it is designed. At a very basic level your EC2 instances can be killed and restarted by amazon at any time. Any application that is run on Amazon EC2 needs to implement clustering to handle this basic situation.

Amazon AWS's EC2 compute instances are not the same as a vps provider. Used properly, it is way more powerful than any other hosting solution. But for most parts, if you are not making simple database backed websites where you are using amazon rds for databases and amazon's elb for load balancing, you will need to spend considerable effort in understanding amazon's architecture and adapting your application for the same.

Dont expect ec2 to behave like your regular shared or dedicated servers, even though they may appear to behave similarly. I recently had a client using another cloud provider suffer major loss of critical data because they deployed on a cloud platform without properly understanding how it works.

As a general rule, unless you are comfortable setting up a clustered environment for your hosting, dont host with amazon aws w/o getting your hosting architecture reviewed by a third party.

I would also recommend against using amazon aws for hosting windows based auto trading apps unless you have handled the scenario where your host can be restarted randomly by amazon.

The same applies to google's compute engine and other cloud based virtual servers. These are not like your traditional VPS.

-- no1lives4ever

Wow , what a detailed reply you have given & also it is really informative. I am from a total non-technical background so that page on Amazon was turning out to be Greek for me & I was in a dilemma as to how should I go for it.

True Data India is also offering VPS @ Rs. 1,000 pm , can you share anything about their service as you have deep understanding about the workings of such systems.

Thanks a lot.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
The Following User Says Thank You to ajrj For This Useful Post:
  #6 (permalink)  
Old 08-07-2016, 11:02 PM
Sunshine's Avatar
Sunshine Sunshine is offline
indiTraders VIP
 
Join Date: Oct 2009
Posts: 2,521
Thanks: 6,169
Thanked 4,425 Times in 1,973 Posts
Sunshine has a reputation beyond repute
Sunshine has a reputation beyond reputeSunshine has a reputation beyond reputeSunshine has a reputation beyond reputeSunshine has a reputation beyond reputeSunshine has a reputation beyond reputeSunshine has a reputation beyond reputeSunshine has a reputation beyond reputeSunshine has a reputation beyond repute
Default

On behalf of many - not so tech savvy- members I request No1 to start a professional service in " trading from the cloud "
__________________
A man who moves a mountain begins by carrying away small stones - Confucius
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
The Following User Says Thank You to Sunshine For This Useful Post:
  #7 (permalink)  
Old 09-07-2016, 01:28 AM
no1lives4ever no1lives4ever is offline
Newbie
 
Join Date: Jul 2009
Posts: 3,295
Thanks: 19,010
Thanked 5,549 Times in 2,358 Posts
no1lives4ever has a reputation beyond repute
no1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond repute
Default

Originally Posted by NJ23 View Post
Thanks no1lives4ever.

Can you elaborate why clustering is not a good idea for a trading virtual server? How does it affect anything? Cause I have not encountered my machine restarting, as in the way we start our laptops and desktops, since I've started using it.

Big whoop! Not a tech person.

The reasoning is a bit long winded, so please be patient with my response below:

Web servers are by nature horizontally scalable. i.e. you can scale a webserver to handle more loads by adding additional web servers to your web server cluster.

The HTTP and DNS protocols used as the backbones of web hosting are designed so that in case a request does not get handled by the server, the browser will retry the request automatically.

This allows one to design servers that can fail and where a failed server's requests can get handled by another server.

So now when you design a web server cluster to handle a large number of requests, you can design it as a cluster of a number of servers, of which a number can fail, and when they fail, their requests are handled by other active servers in the cluster.

In this scenario, you can go with a cheap piece of server hardware which can fail at any point, provided you have a method to exclude the failed hardware from cluster that is serving requests for the website.

This forms the basis of the amazon aws ec2 compute instances and the google compute engine cloud services. You can build a very reliable web site using these instances, provided you design your infrastructure to handle the failure of any single node. Amazon and google have additional design features in their datacenters for reliability. e.g. amazon isolates each datacenter location into multiple availability zones. In addition you can also host your application across multiple datacenters. There are 100s of documents on amazon aws website that detail the architecture of cloud systems and how they are to be designed.

Now coming to trading, the situation is simpler to explain. For automatic trading, you need to have your server designed such that it runs without shutting down or restarting. In case of a shutdown or restart, the system should be designed to automatically restart on the same or another system and then restart with the trading. Doing this on a cloud infrastructure is very difficult and may not be supported by the external interface between your auto trading server and the broker's backend.

Most auto trading solutions are designed as executable programs that run on a windows desktop or server and which do not have any robust mechanism to detect system failures and move the auto trading to another system. This is not a simple problem to solve and most solutions that require such solutions are very costly.

The simple way in which you can normally reduce the risks associated with a restart is to ensure that the system is more reliable. i.e. running on better quality hardware and with redundant power and network cables. But all this adds to the cost. Cloud computing helps cut down on these costs because on a cluster with a large number of compute instances, you will not get any benefit of using more reliable hardware, and in those situations it is better to just assume that hardware can fail and design around that.

Stock exchanges like NSE run on very high end systems which are designed to not stop even when there is a hardware failure. These systems use special hardware and software to ensure the same. Mostly based on the Tandem (now HP) NonStop architecture. https://en.wikipedia.org/wiki/NonStop_(server_computers)

Nowadays you can achieve very similar levels of reliabilty for windows systems using VMWare's fault tolerance feature. https://www.vmware.com/products/vsph...ault-tolerance
This is still a very costly solution for retail traders, but for those looking at completely eliminating hardware issues w/o making major changes to their software design, it is a relatively very cheap solution.

IMO if you are auto trading, then you need to have risk management plan in place.

At the very least, plan for the following:
  • Auto trading system making bad trades and starting to lose money fast. Situations like what happened with Knight Capital.
  • System running the auto trading system goind down due to a hardware failure
  • Network failure - Internet or local lan
  • Failure at the broker's end
  • Software crash

-- no1lives4ever
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
The Following User Says Thank You to no1lives4ever For This Useful Post:
  #8 (permalink)  
Old 09-07-2016, 01:33 AM
no1lives4ever no1lives4ever is offline
Newbie
 
Join Date: Jul 2009
Posts: 3,295
Thanks: 19,010
Thanked 5,549 Times in 2,358 Posts
no1lives4ever has a reputation beyond repute
no1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond reputeno1lives4ever has a reputation beyond repute
Default

This article may help explain some things related to amazon aws better. http://blog.awe.sm/2012/12/18/aws-th...-and-the-ugly/
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
The Following User Says Thank You to no1lives4ever For This Useful Post:
  #9 (permalink)  
Old 09-07-2016, 11:34 AM
NJ23's Avatar
NJ23 NJ23 is offline
Regular
 
Join Date: Jul 2014
Posts: 50
Thanks: 65
Thanked 83 Times in 43 Posts
NJ23 will become famous soon enoughNJ23 will become famous soon enough
Default

Originally Posted by no1lives4ever View Post
The reasoning is a bit long winded, so please be patient with my response below:

Web servers are by nature horizontally scalable. i.e. you can scale a webserver to handle more loads by adding additional web servers to your web server cluster.

The HTTP and DNS protocols used as the backbones of web hosting are designed so that in case a request does not get handled by the server, the browser will retry the request automatically.

This allows one to design servers that can fail and where a failed server's requests can get handled by another server.

So now when you design a web server cluster to handle a large number of requests, you can design it as a cluster of a number of servers, of which a number can fail, and when they fail, their requests are handled by other active servers in the cluster.

In this scenario, you can go with a cheap piece of server hardware which can fail at any point, provided you have a method to exclude the failed hardware from cluster that is serving requests for the website.

This forms the basis of the amazon aws ec2 compute instances and the google compute engine cloud services. You can build a very reliable web site using these instances, provided you design your infrastructure to handle the failure of any single node. Amazon and google have additional design features in their datacenters for reliability. e.g. amazon isolates each datacenter location into multiple availability zones. In addition you can also host your application across multiple datacenters. There are 100s of documents on amazon aws website that detail the architecture of cloud systems and how they are to be designed.

Now coming to trading, the situation is simpler to explain. For automatic trading, you need to have your server designed such that it runs without shutting down or restarting. In case of a shutdown or restart, the system should be designed to automatically restart on the same or another system and then restart with the trading. Doing this on a cloud infrastructure is very difficult and may not be supported by the external interface between your auto trading server and the broker's backend.

Most auto trading solutions are designed as executable programs that run on a windows desktop or server and which do not have any robust mechanism to detect system failures and move the auto trading to another system. This is not a simple problem to solve and most solutions that require such solutions are very costly.

The simple way in which you can normally reduce the risks associated with a restart is to ensure that the system is more reliable. i.e. running on better quality hardware and with redundant power and network cables. But all this adds to the cost. Cloud computing helps cut down on these costs because on a cluster with a large number of compute instances, you will not get any benefit of using more reliable hardware, and in those situations it is better to just assume that hardware can fail and design around that.

Stock exchanges like NSE run on very high end systems which are designed to not stop even when there is a hardware failure. These systems use special hardware and software to ensure the same. Mostly based on the Tandem (now HP) NonStop architecture. https://en.wikipedia.org/wiki/NonStop_(server_computers)

Nowadays you can achieve very similar levels of reliabilty for windows systems using VMWare's fault tolerance feature. https://www.vmware.com/products/vsph...ault-tolerance
This is still a very costly solution for retail traders, but for those looking at completely eliminating hardware issues w/o making major changes to their software design, it is a relatively very cheap solution.

IMO if you are auto trading, then you need to have risk management plan in place.

At the very least, plan for the following:
  • Auto trading system making bad trades and starting to lose money fast. Situations like what happened with Knight Capital.
  • System running the auto trading system goind down due to a hardware failure
  • Network failure - Internet or local lan
  • Failure at the broker's end
  • Software crash

-- no1lives4ever

Right! Got the whole thing now. Thanks matey! The link is very informative.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
The Following User Says Thank You to NJ23 For This Useful Post:
Reply

indiTraders - Forum for the Active Indian Trader > Technical > Auto & Algo Trading


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On



All times are GMT +5.5. The time now is 12:12 PM.


vBulletin Copyright by vBulletin

Content Relevant URLs by vBSEO 3.3.2