I was using Amazon cloud services for past some time. Amazon is one of the amazing service provider and mean time many restrictions and limitations when you are looking for flexible cloud infra and services. On my experience on Amazon EC2, and ELBs; I have noticed some issues which may be very difficult to identify.
ELBs behave very strange some time when servers go offline. As Expected Health checks in ELBs, will manage EC2 downtime as explained. But I have noticed, once all servers in ELB goes down for maintenance or for any rezone it will be marked as Out of service in ELB page.
Yes, this looks normal. The issue comes when these servers come up; at this moment ELB behaves very strange. ELB will show servers are in service after health checks, and even you can see health checks are hitting all servers. But if you try to access ELB URL it will respond like unavailable or it will say service is down.
So surprised rite! Yes this what issue I have noticed. The solution for this was remove the server from ELB and add it back. When I checked with some AWS experts, they said it will come up and will be live but if it’s not coming up remove servers and add it back.
So the conclusion is keep sharp eye on it using monitoring tools. Hop this will be helpful for some one who will be facing similar king of issues with ELBs. Thank you, will come with new topic soon.