Wednesday, August 14, 2013

Centralized logging for distributed applications with pyzmq

Simpler distributed applications can take advantage of centralized logging. PyZMQ, a Python bindings for ØMQ provides log handlers for the python logging module and can be easily used for this purpose. Log handlers utilizes ØMQ Pub/Sub pattern and broadcasts log messages through a PUB socket. It is quite easy to construct the message collector and write messages to a central location.
         +-------------+
         |Machine1:App1+-------------------------
         +-------------+                        |
                                         +---------------+
         +-------------..................|Machine3:Logger|
         |Machine1:App2|                 +---------------+
         +-------------+                          |
                                                  |
                   +-------------+                |
                   |Machine2:App1|-----------------
                   +-------------+



Client Application
To start with, we will need pyzmq library and support for logging library.
import logging
import random
import time
import zmq
from zmq.log.handlers import PUBHandler
Useful format that identifies where the logs are emanating from.

LOG_LEVELS = (logging.DEBUG, logging.INFO, logging.WARN, logging.ERROR, logging.CRITICAL)

formatters = {
        logging.DEBUG: logging.Formatter("%(filename)s:%(lineno)d | %(message)s\n"),
        logging.INFO: logging.Formatter("%(filename)s:%(lineno)d | %(message)s\n"),
        logging.WARN: logging.Formatter("%(filename)s:%(lineno)d | %(message)s\n"),
        logging.ERROR: logging.Formatter("%(filename)s:%(lineno)d | %(message)s\n"),
        logging.CRITICAL: logging.Formatter("%(filename)s:%(lineno)d | %(message)s\n")
        }
interval = 1
port = 5558
And finally the log handler that allows publication of messages over a PUB zmq socket.
ctx = zmq.Context()
pub = ctx.socket(zmq.PUB)
pub.connect('tcp://127.0.0.1:%i' % port)
logger = logging.getLogger("clientapp1")
logger.setLevel(level)
handler = PUBHandler(pub)
handler.formatters = formatters
logger.addHandler(handler)
while True:
        level = random.choice(LOG_LEVELS)
        logger.log(level, "subtopic.subsub::Hello from %i" % os.getpid())
        time.sleep(interval)

You may have also notice the use of specific style of message that helps you provide a specific subtopic which is useful for logging structure. Finally, we will implement the centralized logger.

Centralized logger
import zmq.green as zmq
import logging
import logging.handlers

LOG_LEVELS = {'DEBUG': logging.DEBUG,
              'INFO': logging.INFO,
              'WARN': logging.WARN,
              'ERROR': logging.ERROR,
              'CRITICAL': logging.CRITICAL
              }
port = 5558

The centralized logger implements the SUB pattern (of PUB/SUB) to subscribe to published messages and log the messages to a file. The published messages could emanate from different applications on different machines and provides for centralized logging.
              
logger = logging.getLogger()
context = zmq.Context(context)
socket_fd = context.socket(zmq.SUB)
socket_fd.bind("tcp://localhost:%s" % port)
socket_fd.setsockopt(zmq.SUBSCRIBE, "")
filehandler = logging.handlers.TimedRotatingFileHandler('log file', 'midnight',1)
logger.setLevel(logging.DEBUG)
filehandler.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s | %(levelname)s | %(message)s')
filehandler.setFormatter(formatter)
logger.addHandler(filehandler)
while True:
        topic, message = socket_fd.recv_multipart()
        pos = topic.find('.')
        level = topic
        if pos > 0: level = topic[:pos]
        if message.endswith('\n'): message = message[:-1]
        log_msg = getattr(logging, level.lower())
        if pos > 0: message = topic[pos+1:] + " | " + message
        log_msg(message)

54 comments:

  1. Replies
    1. An important building block of SEO is to develop your website which can be easily understandable for both Online Visitors (most important) and search engine robots. In the current digital age, seo is important for your online success means to generate leads.

      SEO Services in IndiaSEO Company in India SEO Company in India SEO Services in India

      SEO Company in India SEO Services in India

      Delete
  2. google: zeromq logger python
    got me here - that is a nice pattern

    ReplyDelete
  3. Wonderful article with useful and clear explanation. I really enjoyed by reading your blog post. Thanks a lot for sharing this amazing content with us.

    Hadoop Training in Chennai

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. the blog is very useful, interesting and informative. thank you for sharing the blog with us. keep on updating.
    INFORMATICA TRAINING IN CHENNAI

    ReplyDelete
  6. thanks for sharing about coding with screen shot explanation.its very useful for us keep sharing.
    ccna Training in Chennai

    ReplyDelete
  7. the blog is very interesting and will be much useful for us. thank you for sharing the blog with us. please keep on updating.

    SAS TRAINING IN CHENNAII

    ReplyDelete
  8. That is very interesting; you are a very skilled blogger. I have shared your website in my social networks! A very nice guide. I will definitely follow these tips. Thank you for sharing such detailed article.


    SAP training in Chennai

    ReplyDelete
  9. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care.As always, we appreciate your confidence and trust in us..SAP Training in chennai

    ReplyDelete



  10. This content creates a new hope and inspiration with in me. Thanks for sharing article like this. The way you have stated everything above is quite awesome. Keep blogging like this. Thanks.


    SAP training in Chennai

    ReplyDelete
  11. This content creates a new hope and inspiration with in me. Thanks for sharing article like this. The way you have stated everything above is quite awesome. Keep blogging like this. Thanks.


    SAP training in Chennai

    ReplyDelete
  12. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence and trust in us.

    SAP training in Chennai

    ReplyDelete
  13. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence and trust in us.

    SAP training in Chennai

    ReplyDelete

  14. That is very interesting; you are a very skilled blogger. I have shared your website in my social networks!


    SAP training in Chennai

    ReplyDelete
  15. Great and useful article. Creating content regularly is very tough. Your points are motivated me to move on.


    SEO Company in Chennai

    ReplyDelete
  16. Great articles, first of all Thanks for writing such lovely Post! Earlier I thought that posts are the only most important thing on any blog. But here at Shoutmeloud I found how important other elements are for your blog.Keep update more posts..

    Corporate Training in Chennai


    ReplyDelete
  17. Great article thanks!
    It really helped me to establish centralized logging from multiple VMs.

    One thing to point out, if your publishers are running on multiple machines (which actually means different IPs), then subscriber needs to listen for all of them and not only localhost:
    socket_fd.bind("tcp://*:%s" % port)

    ReplyDelete

  18. Thank you for sharing such a nice and interesting blog with us. Hope it might be much useful for us. keep on updating...!!
    seo company in india
    digital marketing company in india
    seo company in chennai
    digital marketing company in chennai

    ReplyDelete
  19. I just see the post i am so happy the post of information's.So I have really enjoyed and reading your blogs for these posts.Any way I’ll be subscribing to your feed and I hope you post again soon.
    SEO Company in Chennai

    ReplyDelete
  20. Usually I do not read post on blogs, but I would like to say that this write-up very forced me to try and do it! Your writing style has been surprised me. Great work admin.Keep update more blog.
    best GMAT coaching classes in chennai

    ReplyDelete
  21. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us. expecting for your..
    SEO Company in India

    ReplyDelete
  22. This information is impressive; I am inspired with your post writing style & how continuously you describe this topic.

    Payday loans in Alabama
    Title loans in South Carolina

    ReplyDelete

  23. Wonderful blog.. Thanks for sharing informative Post. Its very useful to me.

    Installment loans
    Payday loans
    Title loans

    ReplyDelete
  24. i really likes your blog and You have shared the whole concept really well. and Very beautifully
    บาคาร่าออนไลน์
    gclubwritten,
    soulful read! thanks for sharing.
    GCLUB มือถือ

    ReplyDelete

  25. Its a wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much

    Personal Installment Loans
    Payday Cash Advance loan
    Title Car loan
    Cash Advance Loan

    ReplyDelete
  26. you are posting a good information for people and keep maintin it and give more updates too.
    seo company india

    ReplyDelete


  27. This information is impressive; I am inspired with your post writing style & how continuously you describe this topic.


    Pawn Shop

    Pawn Loans

    Pawn Shops

    Pawn Loan

    Pawn Shop near me

    ReplyDelete

  28. This information is impressive; I am inspired with your post writing style & how continuously you describe this topic.


    Pawn Shop

    Pawn Loans

    Pawn Shops

    Pawn Loan

    Pawn Shop near me

    ReplyDelete
  29. very informative, good to understand easily...

    SEO Company in Chennai

    ReplyDelete

  30. Its a wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much

    Installment Loans Near Me
    Title loans Near Me
    Cash Advances Near Me

    ReplyDelete
  31. That was a very good information. Thanks for the program.

    Please click on the link below.
    Advanced Excel Classes in Bangalore

    ReplyDelete
  32. It is really a great and useful piece of info. I’m glad that you shared this helpful info with us. Please keep us informed like this. Thank you for sharing.
    Seo Company in Chennai
    Digital Marketing Company in Chennai

    ReplyDelete
  33. AWS Training in Bangalore - Live Online & Classroom
    myTectra Amazon Web Services (AWS) certification training helps you to gain real time hands on experience on AWS. myTectra offers AWS training in Bangalore using classroom and AWS Online Training globally. AWS Training at myTectra delivered by the experienced professional who has atleast 4 years of relavent AWS experince and overall 8-15 years of IT experience. myTectra Offers AWS Training since 2013 and retained the positions of Top AWS Training Company in Bangalore and India.

    IOT Training in Bangalore - Live Online & Classroom
    IOT Training course observes iot as the platform for networking of different devices on the internet and their inter related communication. Reading data through the sensors and processing it with applications sitting in the cloud and thereafter passing the processed data to generate different kind of output is the motive of the complete curricula. Students are made to understand the type of input devices and communications among the devices in a wireless media.

    ReplyDelete
  34. This is an awesome post.Really very informative and creative contents. These concept is a good way to enhance the knowledge.I like it and help me to development very well.Thank you for this brief explanation and very nice information.Well, got a good knowledge.

    rpa Training in tambaram

    blueprism Training in tambaram

    automation anywhere training in tambaram

    iot Training in tambaram

    rpa training in sholinganallur

    blue prism training in sholinganallur

    automation anywhere training in sholinganallur

    iot training in sholinganallur

    ReplyDelete
  35. All are saying the same thing repeatedly, but in your blog I had a chance to get some useful and unique information, I love your writing style very much, I would like to suggest your blog in my dude circle, so keep on updates.

    java training in chennai | java training in bangalore

    java online training | java training in pune

    selenium training in chennai

    selenium training in bangalore

    ReplyDelete
  36. Hello! This is my first visit to your blog! We are a team of volunteers and starting a new initiative in a community in the same niche. Your blog provided us useful information to work on. You have done an outstanding job.
    Best AWS Training in Chennai | Amazon Web Services Training in Chennai

    AWS Training in Bangalore | Amazon Web Services Training in Bangalore

    Amazon Web Services Training in Pune | Best AWS Training in Pune

    ReplyDelete
  37. This is ansuperior writing service point that doesn't always sink in within the context of the classroom. In the first superior writing service paragraph you either hook the reader's interest or lose it. Of course your teacher, who's getting paid to teach you how to write an good essay, 

    java training in marathahalli | java training in btm layout

    java training in jayanagar | java training in electronic city

    java training in chennai | java training in USA

    selenium training in chennai

    ReplyDelete
  38. Excellent blog, I wish to share your post with my folks circle. It’s really helped me a lot, so keep sharing post like this
    python training in pune
    python online training
    python training in OMR

    ReplyDelete
  39. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    Devops training in sholinganallur
    Devops training in tambaram
    DevOps online Training

    ReplyDelete
  40. Gaining Python certifications will validate your skills and advance your career.
    python certification

    ReplyDelete
  41. I recently came across your blog and have been reading along. I thought I would leave my first comment.

    Blue Prism Training Course in Pune

    Blue Prism Training Institute in Bangalore

    ReplyDelete