European Journey aboard Epic

Recently (just two weeks ago), my family and I returned home to Canada from an amazing seven day European (Western Mediterranean Sea) cruise aboard Norwegian Cruise Line (NCL) Epic. No, we didn’t win an all-expenses-paid trip to Europe, but rather paid all expenses out of our own pockets :( Good thing that unlike the 1985 comedy film European Vacation, we came back with good memories, not controversies and conflicts.

Here is how Norwegian Epic looked like.
Picture taken from Cannes' Beach

We planned (for the trip) for a while, almost few months, since we purchased our Norwegian Cruise Line (NCL) Epic ticket from Costco Travel, back in March 2017, and the execution went almost flawlessly, except some minor issues here and there . Most of the work went on to devise a plan for shore excursions, because NCL's official excursion rate is way too expensive. We also planned how to stay safe, and secure our cash while in Italy and Spain. A few Travel books suggested that Barcelona, Rome and Naples are
Pickpocketing Hotspots. I bought a Sling Chest Bag from Amazon. I had a hard time finding one, as I needed a sling bag that fits an iPad pro. Fortunately, we didn't have to face any pickpocketing. We found Italian, Spanish and French people very friendly, and helpful.

Note:
  • Make sure to check in online through NCL website at least 48 hours before your cruise departure schedule. 
  • Reserve your on-board entertainment (they quickly run out of seat). If you want, you can start registering for them few months in advance. 
  • Print baggage tags in advance.
  • Plan your shore excursions. You can purchase directly from NCL or plan on your own.
In terms of planning for shore excursions, I read travel books, researched on internet (used sites like https://www.rometoolkit.com/), used information available on NCL website as well as Cruise Critic site. My friend and I tried to devise a detailed plan for each city that we were going to go for shore excursions in a separate spread sheet tab with three alternate options. We put together a check list. We reviewed our plan by all members of both families (total seven people) few times and a final time just a week before flying to Barcelona.

In short, here is how our overall travel plan looked like:
  • Flying to Barcelona from Toronto. 
  • Spend a night in Barcelona. 
  • Take cruise 2nd day from Barcelona.
  • Take shore excursion in each city where ship docks and arrive back to Barcelona after seven days 
  • Fly back to Toronto the same day. 
Below map shows are cruise schedule.
NCL Epic route. Image taken from NCL website.

Flying to Barcelona

We flew on Air Transat. Because of the Baggage Handlers' strike, we had a small delay in Toronto Pearson airport. Good thing, we had two kids belonging to Air Transat Kids club, we were prioritized while boarding the plane in Toronto.

Note: Make sure to register (free) your kid(s) to Kids club, you also get free seat selection and other surprise goodies.

While the service provided by Air Transat was very satisfactory, I didn't like the complementary lunch/dinner sandwich. It was kind of tasteless. However, I enjoyed the complementary glass of wine. Thanks Air Transat! Hope, you'll serve better sandwiches next time :)
Passing through the Spanish custom/immigration was hassle free. I said 'Hola' and handed over our family's passports to the immigration officer, he stamped them and let us go. No baggage delay. Hurrah!

In Barcelona

While we were heading out (of the airport) after collecting our baggage, my friend shouted, "Oh look, you're famous here in Barcelona. Someone is holding your name placard". "Yeah, you guessed it right, my friend", I said and I waved to the person holding my name. I wish I was famous, but it's just that I had pre-purchased (online) airport to hotel taxi/van service from suntransfers.com. They charged EUR 58.87 for a seven seater van. It had good and reliable service which I found as a very satisfactory deal. If we hired a van at the airport, we could have saved around EUR 10.00 or so, but we had to wait in a big line at the airport, negotiate and all that stuff. Our driver was pretty courteous, helpful, friendly and knowledgeable. My friend, with his just-in-time Spanish (I guessed little better than Google translate), was able to acquire a lot of information from the driver.

Finally, we were gliding on the street of great football/soccer city of Barcelona. We were so amazed looking around through our car window, we even didn't notice when we arrived at the hotel. Our driver had to knock us hard to let us know. Our hotel Garbí Mil·lenni (located near the cruise port and centre of the city), was nice. I had booked the junior suite for EUR 189.00.
We were greeted by Marco at reception. He was a nice Spanish fellow with a very business like attitude.

After taking a shower and recharging ourselves, we walked towards the world famous La Rambla street. Yes indeed, we had sometime for walking around and getting ourselves lost as we flew to Barcelona one day ahead of our cruise schedule. Even though, one and half day is not really enough time to visit all the places in the city of Barcelona - a city with rich history and great architecture, it's still better than arriving the same day and rushing to the cruise. So, I recommend anyone taking cruise abroad, get there at least a day or two ahead and avoid last minute rush.
Here, I'm walking on famous La Rambla street.
Special note about La Rambla: My family and I are deeply socked and saddened by the news of  the tragic loss of lives and injuries on La Rambla street caused by the terrorist attack. I’d like to dedicate this post for those innocent souls. I’d like to express my deepest sympathy to all those who have been affected by this terrible act. Read more about La Rambla terrorist attack here.


We had just enough time to tour the city on the Hop On Hop Out (HOHO) bus and we had purchased the ticket online in advance to save money (10% discount if you purchase online) and time. Barcelona has three (green, blue, and red) official HOHO bus routes . The first day, we took red and in the morning of 2nd day, we took blue line tour. HOHO bus seemed like a very convenient option for us;  we could get off at any stop where we wished to take more time to look around and could hop back on at the same stop or another.

Note: Another convenient feature of HOHO bus is ticket validity period. It comes with either 24 or 48 hours option. During that time frame, you can use it any number of times in any of the designated route.
A street view of Barcelona 
(picture taken from the top deck of HOHO bus).
On the 2nd day, since we had to checkout from the hotel before noon and rush to cruise, we thought, we didn't have enough time to walk to La Rambla and take HOHO bus to tour the blue line, we decided to take a taxi instead from our hotel to visit the Antoni Gaudi's famous architectural project Sagrada Familia. The hotel receptionist called the taxi for us and we headed to Sagrada Familia. I paid EUR 16.00 for the trip. Luckily, just after spending sometime at Sagrada Familia, I saw the blue line station right beside the Basilica and we hopped on the HOHO bus and ventured the blue line. 
The blue line along with Sagrada Familia and other historic and architectural objects, include:
-Plaça Catalunya – The main square in the heart of historic city.
-Casa Batlló – The dragon-inspired house designed by Gaudí
-Park Guëll – An unique park from where you can see majestic city views.
-Camp Nou Stadium – The home stadium of FC Barcelona.
Sagrada Família, designed by Antoni Gaudí,
still under construction
(Started 1862 and anticipated completion date of 2026)
Tourist hot spot - La Rambla Casa Milà/La Pedrera, designed by Antoni Gaudí
and built between 1906 and 1910,
(picture taken from the top deck of HOHO bus)
At around 12:00 PM, we checked out of the hotel, and headed to the cruise. We got a seven seater van and the driver charged us EUR 25.00. Not a bad deal! The driver was cheerful and helpful and became our unofficial guide too. On our way to the cruise, I could see a huge ship from a distance, parked at the port. But, my younger daughter was saying that she could not see it. She worriedly said, “how come I can’t see it?". Later she laughed to herself, she was under impression that the huge structure looking like a multi story building was in fact our ship. Norwegian Epic is one of the largest ships. Third largest? As per Wikipedia. It accommodates more than 4000 people on board. See more details on the NCL site.

We were in a long queue of people on hot summer day in Barcelona, desperate to check into  the cruise to have that so called first time (or may be nth time) cruise experience.  Standing in line was a daunting task, but thanks to my friend's little son, we got some priority treatment for our luggage and ourselves.

Note: make sure to print your NCL Cruise luggage tag (you can find it in one of the pages of your NCL eDocument) before hand and attach/staple it into your baggage handle, so that your baggage is checked in without any issue.  For details, see NCL page - packaging for your journey. NCL staff deliver your checked in baggage to your room later that afternoon.

During check in, each of us got a NCL card (Note: it works like your identity while in the ship as well as your cash/credit card), with our room number on it. Children got the one with corner chipped, meant they're not eligible to enter into the adult entertainment zone or buy alcohol etc. It also worked as room key and and to enable the light switch.

Note: It is important that you have your NCL card with you or near you all times. It may be a good idea to buy waterproof cardholder, especially for children as they go into the water (pool, water slides etc) all the time.

On Board

Once checked in and passed through security, we took the escalator to enter the ship. In the entrance of the ship, we were greeted by cheerful NCL staff. And as we walked inside, we were amazed by the fancy carpeting and glittering chandeliers, and of course people from different corners of the world. We took the elevator up to the 11th floor where our ocean view balcony suite was located.  It was a beautiful suite with twin beds, along with sofa bed and just-in-time use ceiling bed. Just imagine a bed that comes out of the ceiling! Immediately my family started calling dibs on who got to sleep there but in the end I won! :) Yeah, amazing! That night, while sleeping on my ceiling bed, I could see the ocean in peace and in war/wave. Yeah, I have to admit that initially we didn't know about the existence of ceiling bed in our suite. I became furious, because there was not enough beds for a family of four and went hurriedly to customer service. They calmly explained me that the room service would come in the evening to prepare the ceiling and sofa bed.

Note: It is specially important to choose your room carefully while booking your cruise vacation. Location matters. It will be too noisy if it's near to engine floor or theatre or near to other entertainment venues. Also important is to choose which side of the ship you want your cabin, specially if you have ocean view window or balcony. Some info from NCL here. And some valuable insight from cruisecritc here.

The first day, we familiarized ourselves to all the cruise amenities - swimming pool, water slides, sports zone, gym and fitness place, running track, theatre, casinos, restaurants, buffet etc. We had good lunch in garden caffe on the top floor of the ship beside the swimming pool and water slides. I also spent some time in pool and went in the Jacuzzi to relax myself.

The 2nd day, I woke up early in the morning. I wanted to start with Yoga. I went to the balcony; it was very early, just the beginning of dawn and little bit chilly outside, but peaceful. And guess what? I did my Yoga for the first time in the middle of Mediterranean Sea. Below are two pictures taken from balcony that morning. One early morning and another little later. 
Cargo ship seen far.
Picture taken from my balcony, early morning.
Picture taken from balcony - Mediterranean Sea
I was also excited about jogging/running that day and went to deck 7 for jogging, but I was little bit disappointed that the track was only on one side of the ship, and all the life boats hanging there blocked the ocean view. And also because of hot/warm air coming from all big machineries there, made jogging not so pleasant. See some messages from people regarding jogging track here. Anyways, I ran around 20 minutes.
The whole 2nd day, we were on the water as there was no planned docking and most of us did whatever we wanted. We had continental breakfast at Garden Caffe on the top of the ship. The cafeteria was huge and selection of breakfast items was amazing and it operated 24 hours a day. Later, I went to swimming pool for a bit and then went to jacuzzi, there I talked to lady from Netherland and her husband for few minutes and after that spent time under the sun on beach chairs while reading the book I brought with me - The Millionaire Next Door by Thomas J. Stanley Ph.D., William D. Danko. And, What the heck! I was allegedly imagining myself as millionaire while relaxing and reading the book. Once, I'm awaken from my millionaire day dream, I went to the gym with my friend. When I looked out the window while running on the treadmill it felt like I was running on the ocean.
The Evening was total blast that day. We went to the Epic theatre to watch the 'Burn the Floor' performance. Wow, it was amazing. They presented it in a real professional way. As NCL puts it in their website, "...The performances of Burn the Floor are not just hot - they sizzle..." Yeah, they did. Everybody liked it including my daughters.


Dawn in Naples (Italian: Napoli)

What an extraordinary morning in Napoli! I just awoke up and saw the beginning of the twilight through the glass door (obviously, I was peering from my ceiling bed). I rushed to balcony, and in few minutes I saw the majestic sun rise. And after some time, we were nearing to port of Naples.
Sun rise in Naples (picture taken from the balcony of my suite) City of Napoli from harbour front side
(picture taken from the balcony of my suite)
Today, we were having an early breakfast around 7:15 AM, so that we'd be ready to go out for the excursion. By 8:30 AM, we were ready to get out of the ship.

Note: You need your cruise card to check in and check out. In NCL Epic, the exit deck is usually either deck 6 or deck 4, depending upon the port and their infrastructure. Listen for an announcement to find which deck to exit from.
Note: At Naples, cruise ships dock at Naples Stazione Marittima, which is right in front of the city centre. From here, you can easily walk around the city or take the HOHO bus. Another option is to take the train (Naples Circumvesuviana) and go to Mt. Vesuvius.

We entertained the 2nd option and took the HOHO bus- "City SightSeeing Napali". It was quite a good choice and we really enjoyed the whole trip. The whole Naples city itself looked like a big open museum with it's architecture, arts, monuments and cathedrals.

Note: Just like in Barcelona, Naples standard "City Sightseening" HOHO buses also operate in 3 routes. Line A (Red), Line B (Blue), and Line C (Green). Price: EUR 23.00/adult; EUR 11.5/child. In addition, there is a 4th route (Purple line), which is called "City Sightseeing Hystorical Center - Cathedral" and it's cost is EUR 12.00/person. It operates in small van as it passes through narrow streets neighbourhoods. If you have standard ticket and you also want to take Purple line tour, you can just add EUR 3.00/person.

We ventured through all three tour lines and also the purple one. One of my favourites was the Purple line which passed through city's historic narrow street neighbourhood. Since, big double decker bus would not pass through the narrow street, they'd use smaller fifteen seater van on this route. 

Largo Castello at the Piazza Municipio, Naples
(picture taken from the top deck of HOHO bus)
Guglia of the Immaculate Virgin (1750), Gesu Nuovo square
(picture taken from the top deck of HOHO bus)
Naples, Mt. Vesuvius seen in the background
(Picture taken from the upper deck of HOHO bus. 
One of the historic narrow streets neighbourhood in Naples
(picture taken through the window of HOHO bus)

We returned to the ship after our Naples excursion around 2:30 PM and had fun on the ship. My daughter went for rock climbing, and she successfully touched and ringed the bell. Bravo!
Later in the night, we went again to the Epic theatre to watch a Broadway musical show "Priscilla Queen of the Desert". It featured the journey of three "divas" with hundreds of colourful costumes and headdresses along with disco hits from the 70s and 80s.

Note: Parental guidance is recommended if you're taking children to the show.

Road to Rome

We arrived in Civitavecchia, a port town and comune of the Metropolitan City of Rome on the 3rd day of our cruise trip. We were planning to go to Rome and visit Vatican today, so we were ready and got out by 8:30 AM. We took the free shuttle provided by NCL from the pier to the port gate and from there took a city bus (EUR 2.00 per person) to reach the train station. We got a combined train ticket ( Civitavecchia to Roma Termini (Rome)) plus HOHO bus tickets for Rome city tour at Civitavecchia train station.

Note: Some of the agents working inside Civitavecchia look and act like Train Italia staff, but be careful, they may represent private tour operator.
Note: There are express as well as local commuter train available from Civitavecchia to Rome. For details visit Train Italia web site.

So, we took the train. While going to Rome, the train was jam packed, and not everyone of us got a seat. But we were still happy that we were visiting the historic city of Rome. Once we got out of the Roma Termini, a bunch of street agents (representing tour buses and others) were behind us trying to sell tickets. Since, we had our ticket already purchased, we just moved forward, kind of ignoring them. We had to walk around 7 minutes to reach to our Big Bus Tour bus.

Note: There are multiple HOHO buses operate in Rome. You can find information about all these here. Compare and take the one that best suits you.


We were on the top of a double decker HOHO bus and it seemed like one of the hottest day here in Rome, however, we weren't worried much as we were sight seeing one of the greatest city on earth - Rome; and travel through it's history and visit the places where the ancient Romans lived, worked and socialized. We got a chance to admire the fine arts and architecture attached to Rome's grand buildings and beautiful churches. Compared to Naples, I found that historic monuments and buildings were well preserved/maintained here in Rome. Maybe Rome is richer and gets a bigger budget?
Touring Rome in style? Definitely, next time. 
Rolling Rome on Sageway
monument dedicated to King Victor Emmanuel II,
the first king of Italy, in Piazza Venezia
(picture taken from the top deck of HOHO bus)
Rome colosseum Street of Rome (picture taken from the top deck of HOHO bus)
Road to Vatican. We were entering one of the smallest
but historically significant country of the world.
St. Peter's Basilica and Square, Vatican.
Hoping to see the Pope Francis and get blessings in-person, we spent around 2 hours around Vatican, but he decided to bless us remotely. :) Blessed, we returned back to our HOHO bus to complete the remaining tour of Rome. At around 2:00 PM, we returned back to Roma Termini to take train back to Civitavecchia. We had to ask several people to find out our exact platform, finally, Train Italia's agent helped us figure it out.

Back to the ship, after having a quick late lunch, I went to spend some time in pool and later on in Jacuzzi. That night, we went to watch the musical show "The Epic Beatles" in the Epic Theatre. Someone has posted a footage of the show here in Youtube.
Later on, we decided to go for dinner at the Manhattan dinning room. NCL Epic has a lot of dinning options. See here for details.

Note: If you like Indian dinning, you need to ask for an Indian menu. The main menu does not include Indian dishes.


Touching the Tower in Pisa.

The ship docked in the port of Livorno, Italy at around 7:00 AM and we were out for our excursion at around 9:00 AM. We planned to go to Pisa to see the leaning tower.

Note: There are few option for self excursion here in Livorno.
1. Take train, taxi or tour bus and go to Florence
2. Take train, taxi or tour bus and go to Pisa.
3. Take NCL Shuttle which charges EUR 12.00 and go to downtown Livorno and either take HOHO bus or other form of transportation to go around the city.

We wanted to go to Pisa, and fortunately, found a good deal with a tour bus. We had paid EUR 21.00 per person. The tour bus took us first to downtown Livorno, and from there another bus took us to Pisa.

Note: Here is their website, if you like to get more info or plan ahead for your tour. http://www.livorno.city-sightseeing.it/eng/tour-livorno-pisa.html

Port of Livorno, Italy Piazza dei Miracoli -
Pisa Cathedral with the Leaning Tower of Pisa
For you to guess.
Hey look, I touched the tower!
Note: There is almost a 20 minutes walk from the bus station to the Piazza dei Micacoli. You pass through the parking lot, cross the rail line and go through the beautiful street with a lot of gift shops. Your bargain skill really helps here. Specially, if you want to buy any gift items directly from the walking street vendor.


We spent around 2 hours in Pisa and took bus back to Livorno at 1:00 PM. We were back to the ship by 2:30 PM for lunch. After lunch my friend and I went to try the rock climbing. You probably guessed it right, we couldn't touch and ring the bell, but nevertheless it was a very good experience.
Later that night, we were supposed to enjoy the Cirque Dreams and Dinner Entertainment. All dressed up, we went to the show, but after inquiry we found that the ticket we purchased "standard seating", might not give us the right experience, and also the show was not what we thought - a full fledged circus kind of show, but it seemed more of an acrobatic performance only. We asked to cancel our booking, and a nice lady coordinating the show agreed to give us full refund. We were little disappointed, but found another interesting dance show that featured dancing with the NCL staff.

Coming to Cannes.

The ship arrived around 8:00 AM in Cannes, France, but didn't exactly dock, it remained offshore and tendered the passengers to the shore . Since we had not reserved it in advance, we had to stay in line of hundreds of people. Fortunately, line moved quickly and we arrived at shore in no time.

Note: You can reserve (free) Tender seat from NCL concierge service few days in advance.
Note: On the shore, there is an information kiosk, wash room, wi-fi etc.

At the information desk, my friend asked some questions in French, but to our amazement, the lady replied in English. Any ways, we got a city map, and we headed to take the trip on Le Petit Train.

Note: Le Petit Train starts from Palais des Festivals, the Hotel Majestic beaches and for one hour goes around the historical part of the city and covers the Croisette Tour - luxury hotels, beautiful beaches, brand name boutiques, lavish yachts, Palm Beach, casinos etc. Cost EUR 10.00 (adult), EUR 5.00 (children)

We were amazed by the beauty of Cannes - clean well maintained streets full of flowers, towering palm trees and sandy beaches. It gave us whole different perspective of Europe - Luxurious, and Lavish life style and beauty. I've never seen so many luxury cars - Bentley, Maserati, Ferrari, Porsche, Mercedes, Rolls Royce, Lamborghini you name it, parked in the small fragment of the street. Same goes to brand name boutique stores one after another. No wonder, this city is called sister city of Beverly Hills. If I had a million dollars, I'd definitely go there again, you know, to spend it in style!  
We spent an hour going around the city on Le Petit train, while inhaling fresh Cannes' air and submerging our eyes onto the beauty of French Riviera.
Cannes. Picture taken from balcony of my suite. Le Petit Train
Street of Cannes - see the palm trees (picture taken from the top deck of HOHO bus)
After the tour on Le Petit train, we walked for a while by the sea shore, looking for the perfect place for a blue water beach experience and we found one. Wow! What a pristine and warm water and perfect weather! Beaches in Cannes are really well maintained. They even have a fresh water tap there right on the beach for people to take shower after you are done with salty ocean water. For us, coming from Toronto, where we used to go to fresh water beaches, ocean water was whole new experience. Yeah, it felt salty! I know, I shouldn't have drank the water. But, a big wave passed through my mouth, oh sooo salty!!!
Beach, Cannes, France Beach, Cannes, France
After spending two hours or so, we returned back to Tender Boat station to get back on the ship. We had to catch all the ship events as well! We tried to kind of balance between shore excursions and ship events and entertainments.
Back on the ship, that day other than our regular activities like going to gym, swim and Jacuzzi, we went to the “Michelle” musical show. Michelle not only sang and touched our heart but also made us laugh with her humor and funny stories.

Path to Palma De Mallorca

Even though, our ship was supposed to dock only around 1:00 PM today, I woke up early morning, went to the gym and when I came back to my room, everyone was ready to go for breakfast. Since, it was our last night before checkout early the next morning, we wanted to do as much as possible. After breakfast, my younger daughter went for Bungee Trampoline jumping and my older daughter went for Rock Climbing again.

Finally, our ship docked at Palma island port - Estacion Maritima 1 around 1:00 o'clock in the afternoon. We had already taken our lunch and were ready to explore the beautiful Palma island.

Note: Ships dock at the port directly in front of the old town of Palma. Around 10-15 minutes walk and you can pass the terminal and reach the street where you can catch a HOHO bus (Cost: EUR 18.00/adult, EUR 9.00/child)or other form of transportation or walk further ahead to the Palma town. However, in order to go to Mallorca island you need transportation.
Note: Each HOHO bus stop has name and number. The one near to port is number 14 (Estacion Maritima). If you want to go to beach, get off the HOHO bus at stop # 6 (Av. gabriel Alomar i Villalonga (porta des camp)

It was a very hot day and we had to line up for about 30 minutes for the HOHO bus. Once, we were on the top of open double decker bus, because of ocean wind, we didn't feel that hot after all.
Palma Catedral de Santa María de Palma de Mallorca
(picture taken from the top deck of HOHO bus)
We got off the bus at stop #6 and walked to the beach of Palma. Since it was burning hot and we didn't have a beach umbrella, we did a purchase negotiation with a street vendor. He agreed to sell it at EUR 5.00, even though he originally priced it as EUR 15.00, it's because we promised to return it to him once we were done. We put up our umbrella, but it was so hot that we could barely touch the sand with our bare feet, so in no time, we jumped into the ocean water and it felt amazing!
Sea shore street, Palma. Palma beach, near to HOHO bus stop # 6

Comparing Cannes' beach vs. Palma's one, both were amazing. However, water seemed cleaner in Cannes and a lot of weeds here in Palma. Also, unlike Cannes, no fresh water tap was available for beachgoers here in Palma, so we carried our ocean salt with us to the the ship :(
In the evening, we did our packing (as we had to check out early next morning), attached the baggage label and put our check in baggages in the corridor for NCL staff to pickup.
Later that night, we went to watch the four station concert. It was a really good show. Here, I found some footage of the show in an Youtube video.
After the show, rather than going to sleep, I went to Spice H2O - Hot White Party to get the taste of music and dance under the open sky. Spice H2O is adult only zone for having fun with music, dance and with some energizing cocktail-drink. Here, I found some footage of the show on Youtube.

Note: Get your tag for each luggage from NCL concierge and put your name, room number and other relevant info and attach it to each baggage and put your baggage in the corridor just beside your door. Next day, you'll collect them once you check out.

Saying Goodbye :(

The last day, it felt little sad as we were leaving the luxurious cruise life and going back to our every day life. But we had to do it anyways, so we got our last breakfast and headed for check out around 9:00 AM. I met my room service guy (from Indonesia) in the corridor, and gave him some tips and thanked him for his excellent service.
After checkout, we collected our baggages from baggage carousel and headed out to find a seven seater van to airport. It was a big line (few hundred people were already in line) for taxi. After around 40 minutes our turn came, but there was no van (only taxis were there). Now, we either had to split and take two taxis to the airport or wait for a van. After another twenty minutes, we finally got our van. The Spanish driver this time seemed not as friendly. He charged us more than the standard rate. We also ran into little blunder here. Driver asked us what our airport terminal number was. I checked our Air Transat eTicket and there was no terminal number written for return flight from Barcelona. When I said that there was no terminal number in the ticket, the driver said confidently it was Terminal 1. I countered him saying that we landed in terminal 2 while coming to Barcelona. However, he showed his confidence again and said it was for sure terminal 1. We agreed and he took us to terminal 1. We only found while we were already inside the terminal that we were in wrong terminal and had to go to terminal 2. By that time, our taxi already left and we had to take the airport shuttle bus (Note: takes around 20 minutes from terminal 1 to terminal 2) and go to terminal 2. Since we had enough time until our flight, it did not cause any lasting issues.

Note: Check your ticket carefully. Our Air Transat ticket had terminal info for outbound flight (both Toronto and Barcelona), but for Inbound flight, there was no terminal number for Barcelona airport.

At terminal 2, we found that there was a strike going on by the workers handling luggages, which delayed our flight by almost an hour.

We arrived in Toronto at around 6:00 PM safely. No issues at the airport. We were home in Canada! And by the way, the complimentary food provided by Air Transat was good this time. May be, because it was prepared by Spanish chefs? Anyways, thanks Air Transat!

Overall, the trip was quite fun and became a success story. We met amazing people from around the world, saw European culture, arts and architecture with our own eyes. In terms of the cruise, our family’s favourite would be the sky deck with the pool and water slides and beach chairs. We also equally enjoyed and used the gym, the sports court and the library.
Note: In terms of food, basically, the ship has everything (kind of) and best of all there is 24 hour FOOD! Nothing gets better than that. You can go and eat whenever you like, and eat whatever you like. Each day there is something different to eat and there are always special dinners of a certain theme for example “The Taste of Spain” or “Italian Night” or “Seafood extravaganza.” Regardless of the theme there is something for everyone (kind of) to eat and enjoy no matter what food restrictions you have.
Desert from NCL dinning.
NCL provided overall good services. The room was cleaned twice a day, once around noon - when we were out for the excursion and again in the evening - just before dinner. Room service did quite a good job. During the evening service, they created cute towel animals (towel art) and my daughters really liked them.

Towel (Art) animal created by NCL room service staff for our room.

Personally, both Europe and the cruise were the first time experience for myself and my family, and we really enjoyed it. If I had a chance to go again, I'd like to visit Rome and Naples and get a more closer look of the arts and architecture, but my daughters and wife told me that they would go to Cannes, France instead.

Experience Sharing - Docker Datacenter/Mirantis Docker Enterprise

This post details my experience working with Docker Datacenter (DDC)/Mirantis Docker Enterprise - an integrated container management and security solution and now part of Docker Enterprise Edition (EE) offering. Docker EE is a certified solution which is commercially supported. Refer to https://docs.mirantis.com/docker-enterprise/v3.1/dockeree-products/dee-intro.html for more information on Docker EE. I've worked with both production implementation as well as Proof Of Concept (PoC) solution of DDC. This blog post mainly contains my experience while doing PoC.
Obviously, the first step, while building the DDC is to define/design the architecture of DDC. Docker provides a Docker Reference architecture (https://success.docker.com/Architecture/Docker_Reference_Architecture%3A_Docker_EE_Best_Practices_and_Design_Considerations) and it is a good starting point. For this PoC, we will be creating a DDC based on reference architecture by taking a subset of it. Instead of three Universal Control Plane (UCP) nodes, we'll have just one, instead of three Docker Trusted Registry (DTR) nodes, we'll have one and lastly instead of four UCP worker nodes for application, we'll have just two. As an application load balancer, we'll use Dockerized HA-Proxy. It'll be a separate node (not managed by UCP). For this PoC, we'll use CentOS 7.x powered virtual machines created using 'Oracle VirtualBox'. I'm also going to highlight a few tricks and tips associated with VirtualBox.  We'll also create a client Docker node to communicate with DDC components remotely.

Since, we have decided to take the subset of Docker reference architecture and work on it, we can go ahead and start building the infrastructure. For this POC, the entire infrastructure consists of a Windows 10 laptop with 16 GB memory and an Oracle VM VirtualBox.

1. Create Virtual Machines (VMs)

Let's first create a VM with Centos 7.x Linux. Download CentOS 7-1611 VirtualBox image from osboxes.org and create the first node with 2 GB of memory.

Once the VM is ready, make 6 clones of it. For each clone follow the steps below:

1.1 Network setting:

We'll do few things to emulate a static IP for each virtual machine, otherwise the UCP and DTR will experience an issue if IP changes after the installation. See the recommendation from Docker regarding the static IP and hostname here. Enable two network adapter and configure as below:

  • Adapter 1: NAT - to allow the VM (guest) to communicate with the outside world through host computer's network connection.
  • Adapter 2: Host-only Adapter - to allow connection between host and guest. It also helps us to set static IP.
Refer to https://gist.github.com/pjdietz/5768124 for details on how to set this up. One more thing to remember, if you are using CentOS 7, and want to set Permanent static IP, you need to use Interface Configuration file (refer to https://www.centos.org/docs/5/html/Deployment_Guide-en-US/s1-networkscripts-interfaces.html) instead of /etc/network/interfaces. In my case, I used the following:

1.1.1) Identified the interface used for HostOnly Adapter. See below highlighted in yellow:

$>ip a | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
...
    inet 192.168.56.101/24 brd 192.168.56.255 scope global enp0s8
...
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
...

1.1.2) Created file "/etc/sysconfig/network-scripts/ifcfg-enp0s8" on guest. where enp0s8 is the network interface name with content like this:

#/etc/sysconfig/network-scripts/ifcfg-enp0s8
DEVICE=enp0s8
BOOTPROTO=none
ONBOOT=yes
NETWORK=192.168.56.0
NETMASK=255.255.255.0
IPADDR=192.168.56.101
BROADCAST=192.168.56.255
USERCTL=no

Note: make sure to create unique IP for each clone like 192.168.56.101, 192.168.56.102, 192.168.56.103, ... etc.

1.2 Hostname setup

In order to set hostname on CentOS, follow:
#Update the hostname:
#Set hostname for DDC UCP node 

$>sudo hostnamectl set-hostname centosddcucp
# restart the systemd-hostnamed daemon
$>sudo systemctl restart systemd-hostnamed

1.3) [optional] Update /etc/hosts file

For easy access to each node, add the mapping entries in /etc/hosts file of each VM. The following entries are per my configuration.

#/etc/hosts 
192.168.56.101 centosddcucp
192.168.56.102 centosddcdtr01
192.168.56.103 centosddcwrk01
192.168.56.104 centosddcwrk02
192.168.56.105 centosddcclnt
192.168.56.106 centosddchaproxy 
mydockertest.com



2. Install & Configure Commercially Supported (CS) Docker Engine

2.1) Installation:

Official installation document: https://docs.docker.com/engine/installation/linux/centos/#install-using-the-repository
You can install either using the repository or install from a package. Here, we will install using the repository.  To install Docker Enterprise Edition (Docker EE) using the repository, you need to know the Docker EE repository URL associated with your licensed or trial subscription. To get this information:

  • Go to https://store.docker.com/?overlay=subscriptions.
  • Choose Get Details / Setup Instructions within the Docker Enterprise Edition for CentOS section.
  • Copy the URL from the field labeled Copy and paste this URL to download your Edition.
  • set up Docker’s repositories and install from there, for ease of installation and upgrade tasks. This is the recommended approach.

# 2.1.1 Remove any existing Docker repositories (like docker-ce.repo, docker-ee.repo) from /etc/yum.repos.d/.

# 2.1.2 Store your Docker EE repository URL in a yum variable in /etc/yum/vars/. 

# Note: Replace with the URL you noted from your subscription.

$>sudo sh -c 'echo "" > /etc/yum/vars/dockerurl'

# Note: DOCKER-EE-URL looks something like:
# https://storebits.docker.com/ee/centos/sub-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

# Note: I've replaced the actual text with 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' for confidentiality reason.


# 2.1.3 Install required packages:
$>sudo yum install -y yum-utils device-mapper-persistent-data lvm2
...
Updated:
  lvm2.x86_64 7:2.02.166-1.el7_3.4

Dependency Updated:
  device-mapper.x86_64 7:1.02.135-1.el7_3.4              device-mapper-event.x86_64 7:1.02.135-1.el7_3.4         device-mapper-event-libs.x86_64 7:1.02.135-1.el7_3.4
  device-mapper-libs.x86_64 7:1.02.135-1.el7_3.4         lvm2-libs.x86_64 7:2.02.166-1.el7_3.4

Complete!

# 2.1.4 Add stable repository:


$>sudo yum-config-manager --add-repo /docker-ee.repo
Loaded plugins: fastestmirror, langpacks
adding repo from: https://storebits.docker.com/ee/centos/sub-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/docker-ee.repo
grabbing file https://storebits.docker.com/ee/centos/sub-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/docker-ee.repo to /etc/yum.repos.d/docker-ee.repo
repo saved to /etc/yum.repos.d/docker-ee.repo

# 2.1.5 Update Yum package index:


$>sudo yum makecache fast
...
Loading mirror speeds from cached hostfile
 * base: mirror.its.sfu.ca
 * extras: muug.ca
 * updates: muug.ca
Metadata Cache Created

# 2.1.6 Install Docker CS Engine 

$>sudo yum install docker-ee
...
Installed:
  docker-ee.x86_64 0:17.03.2.ee.4-1.el7.centos

Dependency Installed:
  docker-ee-selinux.noarch 0:17.03.2.ee.4-1.el7.centos

# 2.1.7 Add following content (if doesn't exist) or edit (if required) in /etc/docker/daemon.json
   
   {
      "storage-driver": "device-mapper"
   }
 
# 2.1.8 Check the docker service status, enable (if required) and start

$>sudo systemctl status docker
? docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: https://docs.docker.com


$>sudo systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.


$>sudo systemctl start docker


$>sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

# 2.1.9 Manage Docker as non-root user
# 9.1 Add docker group (if doesn't exist)
$>sudo groupadd docker


# 2.1.10) add user to the 'docker' group
$>sudo usermod -aG docker $USER


# logout and login again, and run:


$>docker ps



3. Install & Configure Universal Control Plane (UCP)

UCP is a cluster management solution for Docker Enterprise. In a nutshell, it itself is a containerized application that runs on (CS) Docker Engine and facilitates user interaction (deploy, configure, monitor etc) through (API, Docker CLI, GUI etc) with other containerized applications managed by DDC.

3.1 Prepare for Installation

  1. Start the virtual machine for UCP node (created in step #1. Create Virtual Machines (VMs))
  2. Follow the CS Docker engine installation steps (step #2. Install & Configure Commercially Supported (CS) Docker Engine)
  3. Make sure Docker is running.
  4. Prepare UCP node:

# Get the firewall zones
$>sudo firewall-cmd --get-zones
work drop internal external trusted home dmz public block


# Open the following ports for public zone
tcp_ports="443 80 2376 2377 4789 7946 12376 12379 12380 12381 12382 12383 12384 12385 12386 12387"
# udp_ports="4789 7946"
# sudo firewall-cmd --permanent --zone=public --add-port=${_port}/;
# For example:
$> sudo firewall-cmd --permanent --zone=public --add-port=2376/tcp;
# Once all ports are added, restart the firewall.
$> sudo firewall-cmd --reload;


3.2 Install Docker Universal Control Plane (UCP)


# 3.2.1 Pull UCP image

$>docker pull docker/ucp:latest
latest: Pulling from docker/ucp
709515475419: Pull complete
6beede3f81f7: Pull complete
37a4fec5e659: Pull complete
Digest: sha256:b8c4a162b5ec6224b31be9ec52c772a8ba3f78995f691237365cfa728341e942
Status: Downloaded newer image for docker/ucp:latest

# 3.2.2 Install UCP
Note: 192.168.56.101 is my UCP node:


$>sudo docker run --rm -it --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp install --host-address 192.168.56.101 --interactive
INFO[0000] Verifying your system is compatible with UCP 2.1.4 (10e6c44)
INFO[0000] Your engine version 17.03.2-ee-4, build 1e6d71e (3.10.0-514.el7.x86_64) is compatible
WARN[0000] Your system uses devicemapper.  We can not accurately detect available storage space.  Please make sure you have at least 3.00 GB available in /var/lib/docker
Admin Username: osboxes
Admin Password:
Confirm Admin Password:
INFO[0033] All required images are present

...

INFO[0001] Initializing a new swarm at 192.168.56.101
INFO[0018] Establishing mutual Cluster Root CA with Swarm
...
INFO[0021] Deploying UCP Service
INFO[0085] Installation completed on centosddcucp (node ywkywo08e6dagbe45aprmbhlc)
INFO[0085] UCP Instance ID: IJUU:N6K6:KVJK:W3BO:LXVL:FBB4:RKF5:XNHM:HTQI:TZVL:XFIO:Z253
INFO[0085] UCP Server SSL: SHA-256 Fingerprint=D2:68:F3:...........:BD
INFO[0085] Login to UCP at https://192.168.56.101:443
INFO[0085] Username: osboxes
INFO[0085] Password: (your admin password)

# 3.2.3 Very UCP containers are running:


$>docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED              STATUS                        PORTS                                                                             NAMES
f615d8c1d5ba        docker/ucp-controller:2.1.4   "/bin/controller s..."   55 seconds ago       Up 55 seconds (healthy)       0.0.0.0:443->8080/tcp                                                             ucp-controller
255119a3444e        docker/ucp-swarm:2.1.4        "/bin/swarm manage..."   About a minute ago   Up 56 seconds                 0.0.0.0:2376->2375/tcp                                                            ucp-swarm-manager
08a6e3789fed        docker/ucp-auth:2.1.4         "/usr/local/bin/en..."   About a minute ago   Up 57 seconds (healthy)       0.0.0.0:12386->4443/tcp                                                           ucp-auth-worker
caa4f9b4543a        docker/ucp-metrics:2.1.4      "/bin/entrypoint.s..."   About a minute ago   Up 58 seconds                 0.0.0.0:12387->12387/tcp                                                          ucp-metrics
58fa77c78bb2        docker/ucp-auth:2.1.4         "/usr/local/bin/en..."   About a minute ago   Up 58 seconds                 0.0.0.0:12385->4443/tcp                                                           ucp-auth-api
867f6aec884c        docker/ucp-auth-store:2.1.4   "rethinkdb --bind ..."   About a minute ago   Up About a minute             0.0.0.0:12383-12384->12383-12384/tcp                                              ucp-auth-store
b3e536f9309b        docker/ucp-etcd:2.1.4         "/bin/etcd --data-..."   About a minute ago   Up About a minute (healthy)   2380/tcp, 4001/tcp, 7001/tcp, 0.0.0.0:12380->12380/tcp, 0.0.0.0:12379->2379/tcp   ucp-kv
927cc6a7b5c8        docker/ucp-cfssl:2.1.4        "/bin/ucp-ca serve..."   About a minute ago   Up About a minute             0.0.0.0:12381->12381/tcp                                                          ucp-cluster-root-ca
ef00fe9f7200        docker/ucp-cfssl:2.1.4        "/bin/ucp-ca serve..."   About a minute ago   Up About a minute             0.0.0.0:12382->12382/tcp                                                          ucp-client-root-ca
b56a56aeeddd        docker/ucp-agent:2.1.4        "/bin/ucp-agent pr..."   About a minute ago   Up About a minute             0.0.0.0:12376->2376/tcp                                                           ucp-proxy
403f88c79f46        docker/ucp-agent:2.1.4        "/bin/ucp-agent agent"   About a minute ago   Up About a minute             2376/tcp                                                                          ucp-agent.ywkywo08e6dagbe45aprmbhlc.18vdyg9uqfuepzslnu0uclxzj

# 3.2.4 Apply license

# Note: in order to apply the license, launch the UCP Web UI https://ucp-host:
# and it gives you an option - either to upload the existing license "Upload License" option or "Get free trial or
# purchase license".

$> docker node ls
ID                           HOSTNAME      STATUS  AVAILABILITY  MANAGER STATUS
ywkywo08e6dagbe45aprmbhlc *  centosddcucp  Ready   Active        Leader


That concludes the installation and configuration of UCP. Next is to install the Docker Trusted Registry (DTR).

4. Install & Configure Docker Trusted Registry (DTR)

4.1 Installation steps

  1. Start the virtual machine for DTR node (created in step #1. Create Virtual Machines (VMs))
  2. Follow the CS Docker engine installation steps (step #2. Install & Configure Commercially Supported (CS) Docker Engine)
  3. Make sure Docker is running.
  4. Add this (DTR) node to DDC UCP:
    • Access the UCP Web UI.
    • Click on "+ Add node" link.
    • It shows you command to run from the node. Copy the command, it looks something like:
      docker swarm join --token 192.168.56.101:2377 

      Note: Swarm token looks something like this: SWMTKN-1-28cwz2szulitkrdult2qskn2ehlljyvs6big4oh31hw8l7ez98-f2ohafrl025tlat99f4yxxxxx

      Note: last 4 digits of SWARM-TOKEN are replaced with xxxxx.


    • Run the command from DTR node:
      $>docker swarm join --token \
      SWMTKN-1-28cwz2szulitkrdult2qskn2ehlljyvs6big4oh31hw8l7ez98-f2ohafrl025tlat99f4yfqq4l \ 192.168.56.101:2377

      This node joined a swarm as a worker.
  5. Generate DTR Installation command string from UCP Web UI:
    • Access UCP Web UI.
    • Under Install DTR (in the newer version of UCP, you have to navigate to Admin Settings --> Docker Trusted Registry), click on Install Now, select appropriate selection and it gives you command to copy. Command looks something like:
      docker run -it --rm docker/dtr install --dtr-external-url https://192.168.56.102 \
      --ucp-node centosddcdtr01 --ucp-insecure-tls --ucp-username osboxes \
      --ucp-url https://192.168.56.101

      Note: Where the --ucp-node is the hostname of the UCP node where you want to deploy DTR

      Here is a screen shot that shows DTR installation command string:
  6. Start the installation.
    Note: DTR installation details can be found at https://docs.docker.com/datacenter/dtr/2.2/guides/admin/install/#step-3-install-dtr

    # 1. Pull the latest version of DTR >>docker pull docker/dtr
    # 2. Run installation command: 


    $>docker run -it --rm docker/dtr install --dtr-external-url https://192.168.56.102 \
    --ucp-node centosddcdtr01 --ucp-insecure-tls --ucp-username osboxes \
    --ucp-url https://192.168.56.101

    INFO[0000] Beginning Docker Trusted Registry installation
    ucp-password:
    INFO[0009] Validating UCP cert
    INFO[0009] Connecting to UCP
    INFO[0009] UCP cert validation successful
    INFO[0010] The UCP cluster contains the following nodes: centosddcucp, centosddcdtr01
    INFO[0017] verifying [80 443] ports on centosddcdtr01
    INFO[0000] Validating UCP cert
    INFO[0000] Connecting to UCP
    INFO[0000] UCP cert validation successful
    INFO[0000] Checking if the node is okay to install on
    INFO[0000] Connecting to network: dtr-ol
    INFO[0000] Waiting for phase2 container to be known to the Docker daemon
    INFO[0001] Starting UCP connectivity test
    INFO[0001] UCP connectivity test passed
    INFO[0001] Setting up replica volumes...
    INFO[0001] Creating initial CA certificates
    INFO[0001] Bootstrapping rethink...
    ...
    ...
    INFO[0115] Installation is complete
    INFO[0115] Replica ID is set to: fc27c2f482e5
    INFO[0115] You can use flag '--existing-replica-id fc27c2f482e5' when joining other replicas to your Docker Trusted Registry Cluster

    # 3. Make sure DTR is running:
       In your browser, navigate to the Docker Universal Control Plane web UI, and navigate to the Applications screen. DTR should be listed as an application.

    # 4. Access DTR Web UI:
    https://

Troubleshooting note: if you find your DTR is having some issue and need to remove and re-install, follow this like https://docs.docker.com/datacenter/dtr/2.2/guides/admin/install/uninstall/

# 1. Uninstall DTR:
$>sudo docker run -it --rm docker/dtr destroy --ucp-insecure-tls


INFO[0000] Beginning Docker Trusted Registry replica destroy
ucp-url (The UCP URL including domain and port): https://192.168.56.101:443
ucp-username (The UCP administrator username): osboxes
ucp-password:
INFO[0049] Validating UCP cert
INFO[0049] Connecting to UCP
INFO[0049] UCP cert validation successful
INFO[0049] No replicas found in this cluster. If you are trying to clean up a broken replica, provide its replica ID manually.
Choose a replica to destroy: bd02a612d0c0
INFO[0109] Force removing replica
INFO[0110] Stopping containers
INFO[0110] Removing containers
INFO[0110] Removing volumes
INFO[0110] Replica removed.

# 2. Remove DTR node from UCP
$>docker node ls
$>docker node rm

# Follow the installation steps (4, 5, 6) again.

5. Setup DDC client node

In order to access all DDC nodes (UCP, DTR, Worker) and perform operation remotely, you need to have a Docker client, configured to communicate with DDC securely.

5.1 Installation steps

  1. Start the virtual machine for UCP node (created in step #1. Create Virtual Machines (VMs))
  2. Follow the CS Docker engine installation steps (step #2. Install & Configure Commercially Supported (CS) Docker Engine)
  3. Make sure Docker is running.
  4. Download UCP client certificate bundle from UCP and extract it on client host machine.     
  • Access UCP Web UI and navigate to User Management
  • Click on User
  • Click on « Create a Client Bundle » as shown below in the screen shot:
  1. Configure client so that it can securely connect to UCP:
    # 1. Extract client bundle on Client node:
    $> unzip ucp-bundle-osboxes.zip
    Archive:  ucp-bundle-osboxes.zip
     extracting: ca.pem
     extracting: cert.pem
     extracting: key.pem
     extracting: cert.pub
     extracting: env.ps1
     extracting: env.cmd
     extracting: env.sh
    # 2. load the DDC environment
    $>eval $(<env.sh)
    # 3. Make sure docker is connected to UCP:
    $>docker ps
    # Note: based on your access level, you should see the Docker processes running on UCP, DTR and Worker(s) node(s).








  • Configure client so that it can securely connect to DTR and push/pull images.
    Note: If DTR is using the auto generated self signed cert, your client Docker Engine
    need to configure to trust the certificate presented by DTR, otherwise, you get "x509: certificate signed by unknown authority" error.
    Refer to: https://docs.docker.com/datacenter/dtr/2.1/guides/repos-and-images/#configure-your-host for detail.
    For CentOS, you can install the DTR certificate in the client trust store as follows:
    # 1. Pull the DTR certificate. Here 192.168.56.102 is my DTR node.
    $>sudo curl -k https://192.168.56.102/ca -o /etc/pki/ca-trust/source/anchors/centosddcdtr01.crt
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  2009  100  2009    0     0   8999      0 --:--:-- --:--:-- --:--:--  9049
    # 2. Update CA Trust

    $>sudo update-ca-trust
    # 3. Start the Docker Engine
    $> sudo systemctl restart docker
    # 4. Test the connectivity from client node to DTR node:
    $> docker login 192.168.56.102
    Username: osboxes
    Password:
    Login Succeeded

  • 5.2 Configure Notary client

    By configuring Notary client, you'll be able to sign Docker image(s) with the private keys in your UCP client bundle, trusted by UCP and easily traced back to your user account. Read details here; https://docs.docker.com/datacenter/dtr/2.2/guides/user/access-dtr/configure-your-notary-client/

    # 5.2.1 Download notary
    $>curl -L https://github.com/docker/notary/releases/download/v0.4.3/notary-Linux-amd64 -o notary


      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100   591    0   591    0     0   1184      0 --:--:-- --:--:-- --:--:--  1184
    100 9518k  100 9518k    0     0  3300k      0  0:00:02  0:00:02 --:--:-- 5115k

    # 5.2.2 Give execution permission
    $>chmod +x notary


    # 5.2.3 Move to /usr/bin
    $>sudo mv notary /usr/bin

    # 5.2.4. Import UCP private key into notary key database:
    $> notary key import ./key.pem
    Enter passphrase for new delegation key with ID 4e672ee (tuf_keys):
    Repeat passphrase for new delegation key with ID 4e672ee (tuf_keys):

    # 5.2.5 List key list
    $>notary key list

    ROLE          GUN    KEY ID                                                              LOCATION
    ----          ---    ------                                                              --------
    delegation           4e672ee5f4de7bf132d03554a8f592236ae6054026efc6b01873fc1b45a61dca    /home/osboxes/.docker/trust/private

    # 5.2.6 Configure notary CLI so that it can talk with the Notary server that’s part of DTR
    # There are few ways it can be accomplished. Easiest one is to configure Notary by creating a ~/.notary/config.json file with the following content:

    {
      "trust_dir" : "~/.docker/trust",
      "remote_server": {
        "url": "",
        "root_ca": ""
      }
    }


    # 5.2.7. [optional} Sign image while pushing to DTR:



    Note: By default, CLI does not sign an image while pushing to DTR. In order to sign image while # pushing, set the environment variable DOCKER_CONTENT_TRUST=1

    5.3 Install Docker Compose:

    Docker Compose is a very handy tool that can be used to  define and manage multi-container Docker application(s). For details refer to https://docs.docker.com/compose/overview/
    Note: Docker for Mac, and Windows may already include docker-compose. In order to find out whether the Docker Compose is already installed, just run the the docker-compose --version command.
      $> docker-compose --version
      ash: docker-compose: command not found...

    Install:

    $>sudo curl -L https://github.com/docker/compose/releases/download/1.14.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
    $>chmod +x /usr/local/bin/docker-compose
    $> docker-compose --version
    docker-compose version 1.14.0, build c7bdf9e

    6. Setup Worker node(s):

    Worker node is real work horse in DDC setup where production application runs. Below are installation steps.

    6.1 Installation steps

    1. Start the virtual machine for UCP node (created in step #1. Create Virtual Machines (VMs))
    2. Follow the CS Docker engine installation steps (step #2. Install & Configure Commercially Supported (CS) Docker Engine)
    3. Make sure Docker is running.
    4. Add this (worker) node to DDC UCP:
      • Access the UCP Web UI.
      • Click on "+ Add node" link.
      • It shows you command to run from the node. Copy the command, it looks something like:
        docker swarm join --token 192.168.56.101:2377 
        Note: Swarm token looks something like this: SWMTKN-1-28cwz2szulitkrdult2qskn2ehlljyvs6big4oh31hw8l7ez98-f2ohafrl025tlat99f4yxxxxx
        Note: last 4 digits of SWARM-TOKEN are replaced with xxxxx.

        Note: 
      • Run the command from worker node:
        $>docker swarm join --token \
        SWMTKN-1-28cwz2szulitkrdult2qskn2ehlljyvs6big4oh31hw8l7ez98-f2ohafrl025tlat99f4yfqq4l \ 192.168.56.101:2377

        This node joined a swarm as a worker.
    5. Repeat steps 1 to 4 for each additional worker node
    6. Once all nodes join the swarm, run the following command from client (while connected to UCP) to confirm and list all the nodes:
      $> docker node ls
      ID                           HOSTNAME        STATUS  AVAILABILITY  MANAGER STATUS
      ivczlaqfyjmtvs0xqk6aivy8p    centosddcwrk02  Ready   Active
      usl2otwy3u6hj3vls9r77i45r    centosddcwrk01  Ready   Active
      ywkywo08e6dagbe45aprmbhlc *  centosddcucp    Ready   Active        Leader
      z2cobi7ag2qqsevfjsvye3d19    centosddcdtr01  Ready   Active   
    7. Optional: Put node like DTR or UCP in "drain" mode, where you don't want application container to deploy. Here, we put DTR in "drain" mode
      #command format: docker node update --availability drain
      $> docker node update --availability drain z2cobi7ag2qqsevfjsvye3d19
      z2cobi7ag2qqsevfjsvye3d19
      $> docker node ls
      ID                           HOSTNAME        STATUS  AVAILABILITY  MANAGER STATUS
      ivczlaqfyjmtvs0xqk6aivy8p    centosddcwrk02  Ready   Active
      usl2otwy3u6hj3vls9r77i45r    centosddcwrk01  Ready   Active
      ywkywo08e6dagbe45aprmbhlc *  centosddcucp    Ready   Active        Leader
      z2cobi7ag2qqsevfjsvye3d19    centosddcdtr01  Ready   Drain   
    Here is how node listing of our PoC appears on UCP Web UI:

    UCP node listing


    7. Create Additional User, Access Label and Network:

    7.1 Create additional user, team and permission label as necessary from UCP Web UI.

    Follow Docker documentation (https://docs.docker.com/datacenter/ucp/2.1/guides/admin/manage-users/create-and-manage-users/) to create user, team and permission levels as required. 

    7.2 Create network.

    As per Docker documentation, containers connected to the default bridge network can communicate with each other by IP address. If you want containers to be able to resolve IP addresses by container name, you should use user-defined networks instead. Traditionally, you could link two containers together using the legacy 'docker run --link ...' option, but here, we are going to define a network and attach our containers, so that they can communicate as required. Details can be found here https://docs.docker.com/engine/userguide/networking/#user-defined-networks
    Note: docker links feature has been deprecated in favor of user defined networks as explained here.
    Service discovery in Docker is network-scoped, meaning the embedded DNS functionality in Docker 
    can be used only by containers or tasks using the same network to resolve each other's addresses, so our plan here is to deploy a set of services that can communicate to each other using DNS. 
    Note: If the destination and source container or service are not on the same network, Docker Engine forwards the DNS query to the default DNS server.


    # Create network 
    # From client node, first connect to UCP:
    eval $(<env.sh)

    $> docker network create -d overlay --label com.docker.ucp.access.label="dev" --label com.docker.ucp.mesh.http=true my_hrm_network
    naf8hvyx22n6lsvb4bq43z968

    # Verify the network   
    $> docker network ls
    NETWORK ID          NAME                             DRIVER              SCOPE
    eea917ac864c        centosddcdtr01/bridge            bridge              local
    065f72b45f37        centosddcdtr01/docker_gwbridge   bridge              local
    229f9949f85f        centosddcucp/bridge              bridge              local
    d8a09aed43ae        centosddcucp/docker_gwbridge     bridge              local
    a18d616d5fed        centosddcucp/host                host                local
    fc423b7dc25f        centosddcucp/none                null                local
    o40j6xknr6ax        dtr-ol                           overlay             swarm
    vwtzfva8q8r3        ingress                          overlay             swarm
    naf8hvyx22n6        my_hrm_network                   overlay             swarm
    tbmwjleceolg        ucp-hrm                          overlay             swarm
         

    Note: if you get error Error response from daemon: Error response from daemon: Error response from daemon: rpc error: code = 3 desc = name must be valid as a DNS name component while creating network, check your network name. Make sure it does not contain any dot '.'. Error message itself is little bit confusing. Refer to issue 31772 for details.

    7.3 Enable "HTTP Routing Mesh" if necessary

    • Login to the UCP web Web UI.
    • Navigate to Admin Settings > Routing Mesh.
    • Check Enable HTTP Routing Mesh.
    • Configure the ports for HRM to listen on, with the defaults being 9080 and 9443. The HTTPS port defaults to 8443 so that it doesn't interfere with the default UCP management port (443).
    Note: If it is a NEW network with label '--label com.docker.ucp.mesh.http=true', you need to disable and then re-enable "HTTP Routing Mesh". It can be done through UCP UI:
    • Disable: Admin Settings --> Routing Mesh  --> Uncheck "Enable HTTP routing mesh". Click on Update button.
    • Enable: Admin Settings --> Routing Mesh  --> check "Enable HTTP routing mesh". Click on Update button.

    8. Docker Application Deployment

    8.1 Preparation:

    For this PoC, we're going to build the custom image of Lets-Chat app and deploy using Docker Compose. Here is how our Dockerfile looks like:
    Note: All the steps listed in step 8.x are executed on or from client node.

    8.1.1 Create Dockerfile for lets-chat:

    From sdelements/lets-chat:latest
    CMD (sleep 60; npm start)

    8.1.2 Create lets-chat image using Dockerfile. Run 'docker build ...' command from the same directory where the Dockerfile is located.

    $> docker build -t lets-chat:1.0 .

    Sending build context to Docker daemon 4.608 kB
    Step 1/2 : FROM sdelements/lets-chat:latest
    latest: Pulling from sdelements/lets-chat
    6a5a5368e0c2: Pull complete
    7b9457ec39de: Pull complete
    ...
    ...
    876c39157780: Pull complete
    Digest: sha256:5b923d428176250653530fdac8a9f925043f30c511b77701662d7f8fab74961c
    Status: Downloaded newer image for sdelements/lets-chat:latest
     ---> 296501fb5b70
    Step 2/2 : CMD (sleep 60; npm start)
     ---> Running in 194eb91d5f59
     ---> 14e03b359b1d
    Removing intermediate container 194eb91d5f59
    Successfully built 14e03b359b1d

    8.1.3 Pull the Mongo DB image:


    $> docker pull mongo
    Using default tag: latest
    latest: Pulling from library/mongo
    f5cc0ee7a6f6: Pull complete
    d99b18c5f0ce: Pull complete
    ...
    ...
    72dc91cfe502: Pull complete
    d610498cfcc7: Pull complete
    Digest: sha256:f1ae736ea5f115822cf6fcef6458839d87bdaea06f40b97934ad913ed348f67d
    Status: Downloaded newer image for mongo:latest
       

    8.1.4 Rename/Tag the images as per DTR namespace and Push the images to DTR:
    # Tag lets-chat image
    $> docker tag lets-chat:1.0 192.168.56.102/osboxes/lets-chat:1.0
    # Tag mongo image
    docker tag mongo:latest 192.168.56.102/osboxes/mongo:latest

    # List the images
    $> docker images
    REPOSITORY                           TAG           IMAGE ID            CREATED             SIZE
    192.168.56.102/osboxes/lets-chat     1.0           14e03b359b1d        5 minutes ago       255 MB
    lets-chat                            1.0           14e03b359b1d        5 minutes ago       255 MB
    mongo                                latest        71c101e16e61        6 days ago          358 MB
    192.168.56.102/osboxes/mongo         latest        71c101e16e61        6 days ago          358 MB
    ....


    8.1.5 Push the images to DTR:
    Note: before pushing the image, you need to create "repository" for the images (if one doesn't exist already). Create corresponding repo from DTR Web UI:






    # Login to DTR
    $ docker login 192.168.56.102 -u osboxes -p
    Login Succeeded

    # Push the mongo image to DTR
    $ docker push 192.168.56.102/osboxes/mongo
    The push refers to a repository [192.168.56.102/osboxes/mongo]
    722b5b443860: Pushed
    beaf3a1d24af: Pushed
    ...
    ...
    2589ed7ad668: Pushed
    d08535b0996b: Pushed

    latest: digest: sha256:f1ae736ea5f115822cf6fcef6458839d87bdaea06f40b97934ad913ed348f67d size: 2614

    # Push the lets-chat image to DTR
    $> docker push 192.168.56.102/osboxes/lets-chat
    The push refers to a repository [192.168.56.102/osboxes/lets-chat]
    fb8b4be9b6e6: Pushed
    d3b5bb1c4411: Pushed
    ...
    ...
    b2ac5371e0f2: Pushed
    142a601d9793: Pushed

    1.0: digest: sha256:92842b34263cfb3045cf2f431852bdc4b4dd8f01bc85eb1d0cd34d00888c9bba size: 2418


    8.1.5 Pull the images to all DDC worker nodes where the images will be instantiated into corresponding containers.

    # Connect to UCP.
    # Note: make sure you run the eval command below from the directory where 
    # the client bundle was extracted
    $>eval $(<env.sh)

    #Pull lets-chat
    $> docker pull 192.168.56.102/osboxes/lets-chat:1.0
    centosddcwrk01: Pulling 192.168.56.102/osboxes/lets-chat:1.0... : downloaded
    centosddcucp: Pulling 192.168.56.102/osboxes/lets-chat:1.0... : downloaded
    centosddcwrk02: Pulling 192.168.56.102/osboxes/lets-chat:1.0... : downloaded
    centosddcdtr01: Pulling 192.168.56.102/osboxes/lets-chat:1.0... : downloaded

    # Pull mongo
    $> docker pull 192.168.56.102/osboxes/mongo
    Using default tag: latest
    centosddcwrk01: Pulling 192.168.56.102/osboxes/mongo:latest... : downloaded
    centosddcucp: Pulling 192.168.56.102/osboxes/mongo:latest... : downloaded
    centosddcwrk02: Pulling 192.168.56.102/osboxes/mongo:latest... : downloaded
    centosddcdtr01: Pulling 192.168.56.102/osboxes/mongo:latest... : downloaded

    8.1.6 Create Docker Compose file:

    Here is how our docker-compose.yml looks like:


    version: "3"
    services:
       mongo:
          image: 192.168.56.102/osboxes/mongo:latest
          networks:
             - my_hrm_network
          deploy:
             placement:
                constraints: [node.role == manager]
             restart_policy:
                condition: on-failure
                max_attempts: 3
                window: 60s
             labels:
                - "com.docker.ucp.access.label=dev"
       lets-chat:
          image: 192.168.56.102/osboxes/lets-chat:1.0
          networks:
             - my_hrm_network
          ports:
             - "8080"
          deploy:
             placement:
                constraints: [node.role == worker]
             mode: replicated
             replicas: 4
             restart_policy:
                condition: on-failure
                max_attempts: 3
                window: 60s
             labels:
                - "com.docker.ucp.mesh.http.8080=external_route=http://mydockertest.com:8080,internal_port=8080"
                - "com.docker.ucp.access.label=dev"
    networks:
       my_hrm_network:
          external:
             name: my_hrm_network   

    Few things to notice in the docker-compose.yml above are:

    1. placement constraints: [node.role == manager] for mongo. We are giving instruction to docker to instantiate the mongo container only on the node which has manager role. role can be “worker” or “manager”.
    2. Label: com.docker.ucp.access.label=dev; define access constraint by label. See for details. https://blog.docker.com/2016/03/role-based-access-control-docker-ucp-tutorial/
    3. Label: com.docker.ucp.mesh.http.8080=external_route=http://mydockertest.com:8080,internal_port=8080"; Here lets-chat application is configured for HRM and to accessed using host mydockertest.com on port 8080, which will be our HA-Proxy's host and port. Docker uses DNS for service discovery as services are created. Docker has different built in routing meshes for high availability. HTTP Routing Mesh (HRM) is an application layer routing mesh that routes HTTP traffic based on DNS hostname is part of UCP 2.0.
    4. Also, note that we are not exposing (explicitly) port for mongo, as mongo and lets-chat are in the same network 'my_hrm_network', they will be able to communicate even though, they will be instantiated in different Host (nodes). For lets-chat, the application listens on port 8080, but we are not publishing it explicitly because we are implementing containers with scaling in mind and relaying on Docker HRM. If you publish port explicitly to host (e.g. -p 8080:8080), then it will be an issue if you have to instantiate more than one replica in the same host, because there will be port conflict as only one process can listen into the same port on the same IP. More detail about HRM and service discovery: https://docs.docker.com/engine/swarm/ingress/#configure-an-external-load-balancer and https://docs.docker.com/datacenter/ucp/2.1/guides/admin/configure/use-domain-names-to-access-services/. Good read about service discovery, load balancing and also Swarm, Ingress and HRM: https://success.docker.com/Architecture/Docker_Reference_Architecture%3A_Service_Discovery_and_Load_Balancing_with_Docker_Universal_Control_Plane_(UCP)

     8.2 Deployment:

    8.2.1 Validate docker-compose.yml:

    # Validate docker-compose.yml, run the following command from the same directory 
    # where docker-compose.yml is located.
    $>docker-compose -f docker-compose.yml config

    WARNING: Some services (lets-chat) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.
         

    Note: the above mentioned WARNING is obvious, because if you deploy your service/container using regular 'docker-compose up -d ...' command, it does not support the 'deploy' key.
    In our case, we are going to use 'docker stack deploy ...' instead. So, we can safely ignore the warning.

    8.2.1 Deploy

    # Execute docker stack deploy command using the compose-file 
    $> docker stack deploy --compose-file docker-compose.yml dev_lets-chat
    Creating service dev_lets-chat_lc-mongo
    Creating service dev_lets-chat_lets-chat

    # Verify service(s) are created:
    $> docker stack ls
    NAME           SERVICES
    dev_lets-chat  2

    # See the service details:
    $> docker stack services dev_lets-chat
    ID            NAME                     MODE        REPLICAS  IMAGE
    kib7peniroci  dev_lets-chat_mongo      replicated  1/1       192.168.56.102/osboxes/mongo:latest
    t7s5xpgdxncs  dev_lets-chat_lets-chat  replicated  4/4       192.168.56.102/osboxes/lets-chat:1.0

       

    As you can see one instance of mongo and 4 instances of lets-chat have been created.
    If you want to learn more about stack deployment, refer to https://docs.docker.com/engine/swarm/stack-deploy/#deploy-the-stack-to-the-swarm


    9. Setup HA-Proxy node:

    Here we will have a simple configuration of HA-Proxy just to show the working idea. Refer to HA-Proxy documentation and Docker documentation for HA-Proxy for details.
    Note: for this PoC, we are deploying ha-proxy outside of swarm cluster.

    9.1 Setup steps

    1. Start the virtual machine for HA-Proxy node (created in step #1. Create Virtual Machines (VMs))
    2. Follow the CS Docker engine installation steps (step #2. Install & Configure Commercially Supported (CS) Docker Engine)
    3. Make sure Docker is running.
    4. Prepare the configuration file for HA-Proxy.



    # /etc/haproxy/haproxy.cfg, version 1.7
    global
       maxconn 4096

    defaults
       mode   http
       timeout connect 5000ms
       timeout client 50000ms
       timeout server 50000ms

    frontend http
       bind *:8080
       option http-server-close
       stats uri /haproxy?stats
       default_backend bckendsrvs

    backend bckendsrvs
       balance roundrobin
       server worker1 192.168.56.103:8080 check
       server worker2 192.168.56.104:8080 check

    Few notes about haproxy.cfg above.
    1. Backend connection. We have 4 replicas (2 replica per node), but as you can see only two back-end connections are mentioned in the configuration file. It is the beauty of using Docker Swarm HRM. As long as the traffic reaches to any of the HRM node, whether the actual replica is running or not there, swarm automatically directs traffic to one of the replicas running in one of the available nodes. Docker swarm also takes care of load balancing among all replicas.
    2. The check option at the end of the server directives specifies that health checks should be performed on those back-end servers.
    3. Frontend section defines bind (ip and port) configuration for the proxy and reference to the corresponding backend configuration. In this case, it is listening to all available IPs on port 8080.
    4. 'stats uri' defines the status URI.
    Now, we have our ha-proxy configuration file is ready, let's build the custom ha-proxy image and instantiate it.

    9.2 Create Dockerfile for HA-Proxy

    FROM haproxy:1.7
    COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg

    Note: In this case, haproxy.cfg and Dockerfile are located in the same directory from where we are executing 'docker build ...' command as shown below.

    9.3) Create custom image

    # Create Image

    $> docker build -t my_haproxy:1.7 .
    Sending build context to Docker daemon 3.072 kB
    Step 1/2 : FROM haproxy:1.7
    1.7: Pulling from library/haproxy
    ef0380f84d05: Pull complete
    405e00049647: Pull complete
    c97485231395: Pull complete
    389e4de140a0: Pull complete
    9abb32070ad9: Pull complete
    Digest: sha256:c335ec625d9a9b71fa5269b815597392a9d2418fa1cedb4ae0af17be8029a5b4
    Status: Downloaded newer image for haproxy:1.7
     ---> d66f0c435360
    Step 2/2 : COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
     ---> 182b33ee6345
    Removing intermediate container 4416fbab54be
    Successfully built 182b33ee6345

    # List image
    $> docker images
    REPOSITORY          TAG                 IMAGE ID            CREATED              SIZE
    my_haproxy          1.7                 182b33ee6345        About a minute ago   135 MB
    haproxy             1.7                 d66f0c435360        6 days ago           135 MB


    9.4 Verify the configuration and Instantiate ha-proxy container:

    # Verify the configuration file: 
    $> docker run -it --rm --name haproxy-syntax-check my_haproxy:1.7 haproxy -c \
       -f /usr/local/etc/haproxy/haproxy.cfg

     haproxy-systemd-wrapper: executing /usr/local/sbin/haproxy -p /run/haproxy.pid -c -f /usr/local/etc/haproxy/haproxy.cfg -Ds
    Configuration file is valid


    # Instantiate ha-proxy instance:
    $> docker run -d --name ddchaproxy -p 8080:8080 my_haproxy:1.7


    5bc06e2680e72475f2585c453f6ada0a5ef349e5222f9e75b2c0f98eb1a0462f


    10. Access and Verify Application 

    10.1) Accessing application:
    Once the ha-proxy is running, access the application. Make sure firewall is not blocking the port (that ha-proxy is listening) .

    http://<ha-proxy-host>:<ha-proxy-port>/<application-uri>

    Important: In order to access application, you need to make sure that the '' in the above URL matches the 'host' part of the external-route configuration of HRM.
    In our case it is 'mydockertest.com', so make sure 'mydockertest.com' resolves to the IP address of the ha-proxy. It is the way how HRM along with Swarm discover the services and route the requests in Ingress cluster and we are able to scale containers dynamically.

    10.2) Application verification:

    10.2.1) Get the stat from haproxy. Along with other things, stat shows the request count and which Swarm node is serving the request:
    http://:/haproxy?stats

    10.2.2) First access lets-chat through Web-UI (http://mydockertest.com:8080). Create your account.  Log using your credential. Once you create the account and able to login, in order to verify that the lets-chat is making successful connection to the mongo db, you can do the following:

    # Inspect the lets-chat instance:
    $> docker inspect 3d046c183b6d | grep mongo
       "LCB_DATABASE_URI=mongodb://mongo/letschat",

    #Access the mongodb instance and run mongo shell to verify the data.
    $> docker exec -it jbz7h5hdvb20 bash

    # Launch the mongo shell
    root@jbz7h5hdvb20:/# mongo
    MongoDB shell version v3.4.5
    connecting to: mongodb://127.0.0.1:27017
    MongoDB server version: 3.4.5
    Welcome to the MongoDB shell.

    # Run command 'show dbs' and make sure letschat database is in the list.
    > show dbs
    admin     0.000GB
    letschat  0.000GB
    local     0.000GB

    # Connect to letschat database.
    > use letschat
    switched to db letschat

    # Get the users table and run find and make sure it shows your account data.
    > show collections
    messages
    rooms
    sessions
    usermessages
    users

    # Make sure Users table has account data that was created before.
    > db.users.find()
    { "_id" : ObjectId("595bdce9559bb1000eae7b9e"), "displayName" : "Purna", "lastName" : "Poudel", "firstName" : "Purna", "password" : "$2a$10$JlZrr3Gu3aklxx4qeUK6uuDF3jQDZ/CuA17.Clm6VKk6/NN35QOT6", "email" : "purna.poudel@gmail.com", "username" : "ppoudel", "provider" : "local", "messages" : [ ], "rooms" : [ ], "joined" : ISODate("2017-07-04T18:22:33.868Z"), "__v" : 0 }

    Once you have Docker Datacenter up and running, upgrade it to Docker EE 2.0 and UCP 3.x to have choice of Swarm or Kubernetes orchestration. See my post Upgrade to Docker EE 2.0 and UCP 3.x for Choice of Swarm or Kubernetes Orchestration.


    Looks like you're really into Docker, see my other related blog posts below: