Wednesday, August 14, 2013

Centralized logging for distributed applications with pyzmq

Simpler distributed applications can take advantage of centralized logging. PyZMQ, a Python bindings for ØMQ provides log handlers for the python logging module and can be easily used for this purpose. Log handlers utilizes ØMQ Pub/Sub pattern and broadcasts log messages through a PUB socket. It is quite easy to construct the message collector and write messages to a central location.
         +-------------+
         |Machine1:App1+-------------------------
         +-------------+                        |
                                         +---------------+
         +-------------..................|Machine3:Logger|
         |Machine1:App2|                 +---------------+
         +-------------+                          |
                                                  |
                   +-------------+                |
                   |Machine2:App1|-----------------
                   +-------------+



Client Application
To start with, we will need pyzmq library and support for logging library.
import logging
import random
import time
import zmq
from zmq.log.handlers import PUBHandler
Useful format that identifies where the logs are emanating from.

LOG_LEVELS = (logging.DEBUG, logging.INFO, logging.WARN, logging.ERROR, logging.CRITICAL)

formatters = {
        logging.DEBUG: logging.Formatter("%(filename)s:%(lineno)d | %(message)s\n"),
        logging.INFO: logging.Formatter("%(filename)s:%(lineno)d | %(message)s\n"),
        logging.WARN: logging.Formatter("%(filename)s:%(lineno)d | %(message)s\n"),
        logging.ERROR: logging.Formatter("%(filename)s:%(lineno)d | %(message)s\n"),
        logging.CRITICAL: logging.Formatter("%(filename)s:%(lineno)d | %(message)s\n")
        }
interval = 1
port = 5558
And finally the log handler that allows publication of messages over a PUB zmq socket.
ctx = zmq.Context()
pub = ctx.socket(zmq.PUB)
pub.connect('tcp://127.0.0.1:%i' % port)
logger = logging.getLogger("clientapp1")
logger.setLevel(level)
handler = PUBHandler(pub)
handler.formatters = formatters
logger.addHandler(handler)
while True:
        level = random.choice(LOG_LEVELS)
        logger.log(level, "subtopic.subsub::Hello from %i" % os.getpid())
        time.sleep(interval)

You may have also notice the use of specific style of message that helps you provide a specific subtopic which is useful for logging structure. Finally, we will implement the centralized logger.

Centralized logger
import zmq.green as zmq
import logging
import logging.handlers

LOG_LEVELS = {'DEBUG': logging.DEBUG,
              'INFO': logging.INFO,
              'WARN': logging.WARN,
              'ERROR': logging.ERROR,
              'CRITICAL': logging.CRITICAL
              }
port = 5558

The centralized logger implements the SUB pattern (of PUB/SUB) to subscribe to published messages and log the messages to a file. The published messages could emanate from different applications on different machines and provides for centralized logging.
              
logger = logging.getLogger()
context = zmq.Context(context)
socket_fd = context.socket(zmq.SUB)
socket_fd.bind("tcp://localhost:%s" % port)
socket_fd.setsockopt(zmq.SUBSCRIBE, "")
filehandler = logging.handlers.TimedRotatingFileHandler('log file', 'midnight',1)
logger.setLevel(logging.DEBUG)
filehandler.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s | %(levelname)s | %(message)s')
filehandler.setFormatter(formatter)
logger.addHandler(filehandler)
while True:
        topic, message = socket_fd.recv_multipart()
        pos = topic.find('.')
        level = topic
        if pos > 0: level = topic[:pos]
        if message.endswith('\n'): message = message[:-1]
        log_msg = getattr(logging, level.lower())
        if pos > 0: message = topic[pos+1:] + " | " + message
        log_msg(message)

103 comments:

  1. Replies
    1. An important building block of SEO is to develop your website which can be easily understandable for both Online Visitors (most important) and search engine robots. In the current digital age, seo is important for your online success means to generate leads.

      SEO Services in IndiaSEO Company in India SEO Company in India SEO Services in India

      SEO Company in India SEO Services in India

      Delete
  2. google: zeromq logger python
    got me here - that is a nice pattern

    ReplyDelete
  3. Wonderful article with useful and clear explanation. I really enjoyed by reading your blog post. Thanks a lot for sharing this amazing content with us.

    Hadoop Training in Chennai

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. the blog is very useful, interesting and informative. thank you for sharing the blog with us. keep on updating.
    INFORMATICA TRAINING IN CHENNAI

    ReplyDelete
  6. thanks for sharing about coding with screen shot explanation.its very useful for us keep sharing.
    ccna Training in Chennai

    ReplyDelete
  7. the blog is very interesting and will be much useful for us. thank you for sharing the blog with us. please keep on updating.

    SAS TRAINING IN CHENNAII

    ReplyDelete
  8. That is very interesting; you are a very skilled blogger. I have shared your website in my social networks! A very nice guide. I will definitely follow these tips. Thank you for sharing such detailed article.


    SAP training in Chennai

    ReplyDelete
  9. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care.As always, we appreciate your confidence and trust in us..SAP Training in chennai

    ReplyDelete



  10. This content creates a new hope and inspiration with in me. Thanks for sharing article like this. The way you have stated everything above is quite awesome. Keep blogging like this. Thanks.


    SAP training in Chennai

    ReplyDelete
  11. This content creates a new hope and inspiration with in me. Thanks for sharing article like this. The way you have stated everything above is quite awesome. Keep blogging like this. Thanks.


    SAP training in Chennai

    ReplyDelete
  12. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence and trust in us.

    SAP training in Chennai

    ReplyDelete
  13. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence and trust in us.

    SAP training in Chennai

    ReplyDelete

  14. That is very interesting; you are a very skilled blogger. I have shared your website in my social networks!


    SAP training in Chennai

    ReplyDelete
  15. Great and useful article. Creating content regularly is very tough. Your points are motivated me to move on.


    SEO Company in Chennai

    ReplyDelete
  16. Great articles, first of all Thanks for writing such lovely Post! Earlier I thought that posts are the only most important thing on any blog. But here at Shoutmeloud I found how important other elements are for your blog.Keep update more posts..

    Corporate Training in Chennai


    ReplyDelete
  17. Great article thanks!
    It really helped me to establish centralized logging from multiple VMs.

    One thing to point out, if your publishers are running on multiple machines (which actually means different IPs), then subscriber needs to listen for all of them and not only localhost:
    socket_fd.bind("tcp://*:%s" % port)

    ReplyDelete

  18. Thank you for sharing such a nice and interesting blog with us. Hope it might be much useful for us. keep on updating...!!
    seo company in india
    digital marketing company in india
    seo company in chennai
    digital marketing company in chennai

    ReplyDelete
  19. I just see the post i am so happy the post of information's.So I have really enjoyed and reading your blogs for these posts.Any way I’ll be subscribing to your feed and I hope you post again soon.
    SEO Company in Chennai

    ReplyDelete
  20. Usually I do not read post on blogs, but I would like to say that this write-up very forced me to try and do it! Your writing style has been surprised me. Great work admin.Keep update more blog.
    best GMAT coaching classes in chennai

    ReplyDelete
  21. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us. expecting for your..
    SEO Company in India

    ReplyDelete
  22. This information is impressive; I am inspired with your post writing style & how continuously you describe this topic.

    Payday loans in Alabama
    Title loans in South Carolina

    ReplyDelete

  23. Wonderful blog.. Thanks for sharing informative Post. Its very useful to me.

    Installment loans
    Payday loans
    Title loans

    ReplyDelete
  24. i really likes your blog and You have shared the whole concept really well. and Very beautifully
    บาคาร่าออนไลน์
    gclubwritten,
    soulful read! thanks for sharing.
    GCLUB มือถือ

    ReplyDelete

  25. Its a wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much

    Personal Installment Loans
    Payday Cash Advance loan
    Title Car loan
    Cash Advance Loan

    ReplyDelete
  26. you are posting a good information for people and keep maintin it and give more updates too.
    seo company india

    ReplyDelete


  27. This information is impressive; I am inspired with your post writing style & how continuously you describe this topic.


    Pawn Shop

    Pawn Loans

    Pawn Shops

    Pawn Loan

    Pawn Shop near me

    ReplyDelete

  28. This information is impressive; I am inspired with your post writing style & how continuously you describe this topic.


    Pawn Shop

    Pawn Loans

    Pawn Shops

    Pawn Loan

    Pawn Shop near me

    ReplyDelete
  29. very informative, good to understand easily...

    SEO Company in Chennai

    ReplyDelete

  30. Its a wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much

    Installment Loans Near Me
    Title loans Near Me
    Cash Advances Near Me

    ReplyDelete
  31. That was a very good information. Thanks for the program.

    Please click on the link below.
    Advanced Excel Classes in Bangalore

    ReplyDelete
  32. It is really a great and useful piece of info. I’m glad that you shared this helpful info with us. Please keep us informed like this. Thank you for sharing.
    Seo Company in Chennai
    Digital Marketing Company in Chennai

    ReplyDelete
  33. AWS Training in Bangalore - Live Online & Classroom
    myTectra Amazon Web Services (AWS) certification training helps you to gain real time hands on experience on AWS. myTectra offers AWS training in Bangalore using classroom and AWS Online Training globally. AWS Training at myTectra delivered by the experienced professional who has atleast 4 years of relavent AWS experince and overall 8-15 years of IT experience. myTectra Offers AWS Training since 2013 and retained the positions of Top AWS Training Company in Bangalore and India.

    IOT Training in Bangalore - Live Online & Classroom
    IOT Training course observes iot as the platform for networking of different devices on the internet and their inter related communication. Reading data through the sensors and processing it with applications sitting in the cloud and thereafter passing the processed data to generate different kind of output is the motive of the complete curricula. Students are made to understand the type of input devices and communications among the devices in a wireless media.

    ReplyDelete
  34. This is an awesome post.Really very informative and creative contents. These concept is a good way to enhance the knowledge.I like it and help me to development very well.Thank you for this brief explanation and very nice information.Well, got a good knowledge.

    rpa Training in tambaram

    blueprism Training in tambaram

    automation anywhere training in tambaram

    iot Training in tambaram

    rpa training in sholinganallur

    blue prism training in sholinganallur

    automation anywhere training in sholinganallur

    iot training in sholinganallur

    ReplyDelete
  35. All are saying the same thing repeatedly, but in your blog I had a chance to get some useful and unique information, I love your writing style very much, I would like to suggest your blog in my dude circle, so keep on updates.

    java training in chennai | java training in bangalore

    java online training | java training in pune

    selenium training in chennai

    selenium training in bangalore

    ReplyDelete
  36. Hello! This is my first visit to your blog! We are a team of volunteers and starting a new initiative in a community in the same niche. Your blog provided us useful information to work on. You have done an outstanding job.
    Best AWS Training in Chennai | Amazon Web Services Training in Chennai

    AWS Training in Bangalore | Amazon Web Services Training in Bangalore

    Amazon Web Services Training in Pune | Best AWS Training in Pune

    ReplyDelete
  37. This is ansuperior writing service point that doesn't always sink in within the context of the classroom. In the first superior writing service paragraph you either hook the reader's interest or lose it. Of course your teacher, who's getting paid to teach you how to write an good essay, 

    java training in marathahalli | java training in btm layout

    java training in jayanagar | java training in electronic city

    java training in chennai | java training in USA

    selenium training in chennai

    ReplyDelete
  38. Excellent blog, I wish to share your post with my folks circle. It’s really helped me a lot, so keep sharing post like this
    python training in pune
    python online training
    python training in OMR

    ReplyDelete
  39. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    Devops training in sholinganallur
    Devops training in tambaram
    DevOps online Training

    ReplyDelete
  40. Gaining Python certifications will validate your skills and advance your career.
    python certification

    ReplyDelete
  41. I recently came across your blog and have been reading along. I thought I would leave my first comment.

    Blue Prism Training Course in Pune

    Blue Prism Training Institute in Bangalore

    ReplyDelete
  42. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    Selenium Training in Bangalore | Selenium Training in Bangalore | Selenium Training in Bangalore | Selenium Training in Bangalore

    ReplyDelete
  43. The post is written in very a good manner and it entails many useful information for me. I am happy to find your distinguished way of writing the post. Now you make it easy for me to understand and implement the concept.
    python training in velachery
    python training institute in chennai

    ReplyDelete
  44. I have picked cheery a lot of useful clothes outdated of this amazing blog. I’d love to return greater than and over again. Thanks! 

    angularjs Training in chennai
    angularjs-Training in pune

    angularjs-Training in chennai

    angularjs Training in chennai

    angularjs-Training in tambaram

    ReplyDelete
  45. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    Devops Training in pune

    ReplyDelete
  46. Does your blog have a contact page? I’m having problems locating it but, I’d like to shoot you an email. I’ve got some recommendations for your blog you might be interested in hearing.
    AWS Training in Chennai |Best Amazon Web Services Training in Chennai
    AWS Training in Rajaji Nagar | Amazon Web Services Training in Rajaji Nagar

    Best AWS Training Institute in BTM Layout Bangalore ,AWS Coursesin BTM

    ReplyDelete
  47. Selenium is one of the most popular automated testing tool used to automate various types of applications. Selenium is a package of several testing tools designed in a way for to support and encourage automation testing of functional aspects of web-based applications and a wide range of browsers and platforms and for the same reason, it is referred to as a Suite.

    Selenium Interview Questions and Answers
    Javascript Interview Questions
    Human Resource (HR) Interview Questions

    ReplyDelete
  48. SQream Technologies provides you with a state of the art software which combines modern GPU technology (Graphic Processing Units) with the best practices in today’s Big Data platforms, providing up to 100x faster insights from data.
    Bigdata Training in Chennai OMR

    ReplyDelete
  49. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.

    rpa interview questions and answers
    automation anywhere interview questions and answers
    blueprism interview questions and answers
    uipath interview questions and answers
    rpa training in chennai

    ReplyDelete
  50. I found your blog while searching for the updates, I am happy to be here. Very useful content and also easily understandable providing.. Believe me I did wrote an post about tutorials for beginners with reference of your blog. 
    angularjs online Training

    angularjs Training in marathahalli

    angularjs interview questions and answers

    angularjs Training in bangalore

    angularjs Training in bangalore

    angularjs Training in chennai

    ReplyDelete
  51. Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.
    Java training in Chennai | Java training institute in Chennai | Java course in Chennai

    Java training in USA

    Java training in Bangalore | Java training in Indira nagar

    Java training in Bangalore | Java training in Rajaji nagar

    ReplyDelete
  52. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.

    rpa training in velachery| rpa training in tambaram |rpa training in sholinganallur | rpa training in annanagar| rpa training in kalyannagar

    ReplyDelete
  53. I always enjoy reading quality articles by an individual who is obviously knowledgeable on their chosen subject. Ill be watching this post with much interest. Keep up the great work, I will be back
    online Python training
    python training in chennai

    ReplyDelete
  54. The knowledge of technology you have been sharing thorough this post is very much helpful to develop new idea. here by i also want to share this.
    Best Devops training in sholinganallur
    Devops training in velachery
    Devops training in annanagar
    Devops training in tambaram

    ReplyDelete
  55. Great Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us and I never get bored while reading your article because, they are becomes a more and more interesting from the starting lines until the end.

    angularjs-Training in sholinganallur

    angularjs-Training in velachery

    angularjs Training in bangalore

    angularjs Training in bangalore

    angularjs Training in btm

    ReplyDelete
  56. เผาผลาญไขมัน ไม่มีอันตราย เป็นเทคโนโลยี ที่มีคุณภาพสูงสำหรับการเผาผลาญไขมันส่วนเกิน
    เครื่องจะตรวจจำนวนไขมัน และก็คำนวนพลังงานที่จำเป็นต้องใช้อัตโนมัติ รวมทั้งพวกเราจะใช้เครื่องไม้เครื่องมือยิงคลื่น
    อัลตร้าซาวด์เข้าไปในรอบๆที่ได้ทำเครื่องหมายไว้อย่างสม่ำเสมอ โดยไม่เจ็บต่อเยื่อรอบๆ ใกล้เคียง ไม่ว่าจะเป็นผิวหนังข้างบน เซลล์ประสาท หรือ เส้นโลหิต ต่อจากนั้นเซลล์ไขมันจะถูกแปลงให้เป็นของเหลวแล้วถูกซึมซับไปสู่กระแส เลือด แล้วถูกนำไปเผาผลาญโดยระบบการกำจัดไขมันของ
    ร่างกายตามธรรมชาติ ทำให้ท่านสัมผัสได้ถึงรูปร่างที่กระชับแล้วก็เรียบเนียนขึ้นในทันทีข้างหลังการดูแลรักษา โดยไม่เป็นผลใกล้กันใดๆก็ตามต่อสุขภาพรวมทั้งสามารถดำเนิน
    ชีวิตตามธรรมดา โดยไม่ต้องพักรักษาตัว

    สลายไขมัน กระชับสัดส่วน
    สลายไขมัน ไม่ผ่าตัด

    ReplyDelete

  57. Very nice post here thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's. Best Machine learning training in chennai
    machine learning with python course in Chennai
    machine learning classroom training in chennai

    ReplyDelete
  58. Really great post, I simply unearthed your site and needed to say that I have truly appreciated perusing your blog entries. I want to say thanks for great sharing.

    angularjs Training in bangalore

    angularjs interview questions and answers

    angularjs Training in marathahalli

    angularjs interview questions and answers

    angularjs-Training in pune

    ReplyDelete
  59. hi, nice information is given in this blog. Thanks for sharing this type of information, it is so useful for me. nice work keep it up. best digital marketing company in delhi

    ReplyDelete
  60. I read this post two times, I like it so much, please try to keep posting & Let me introduce other material that may be good for our community.
    Best Devops Training in pune
    Devops interview questions and answers

    ReplyDelete
  61. This is an awesome post.Really very informative and creative contents. These concept is a good way to enhance the knowledge.I like it and help me to development very well.Thank you for this brief explanation and very nice information.Well, got a good knowledge.
    python course in pune
    python course in chennai
    python Training in Bangalore

    ReplyDelete
  62. Laminated Doors manufacturer in hubli
    Thanks for giving great kind of information. So useful and practical for me. Thanks for your excellent blog, nice work keep it up thanks for sharing the knowledge.

    ReplyDelete
  63. led lawn lights in delhi
    Thanks for giving great kind of information. So useful and practical for me. Thanks for your excellent blog, nice work keep it up thanks for sharing the knowledge.

    ReplyDelete
  64. I found your blog while searching for the updates, I am happy to be here. Very useful content and also easily understandable providing.. Believe me I did wrote an post about tutorials for beginners with reference of your blog.

    rpa training in chennai |rpa course in chennai|
    rpa training in bangalore | best rpa training in bangalore | rpa course in bangalore | rpa training institute in bangalore | rpa training in bangalore | rpa online training

    ReplyDelete
  65. Really very nice blog information for this one and more technical skills are improve,i like that kind of post.
    rpa training in chennai |best rpa training in chennai|
    rpa training in bangalore | best rpa training in bangalore

    ReplyDelete