Skip to main content

BIG Data, Hadoop – Chapter 2 - Data Life Cycle

Data Life Cycle

The data life cycle is pictorial defined as show below:

  
As we see, in our current system, we capture/ Extract our data, then we store it and later we process for reporting and analytics.

But in case of big data, the problem lies in storing and then processing it faster. Hence Hadoop takes this portion, where it stores the data in effective format (Hadoop distributed File System) and also process using its engine (Map Reduce Engine).




Since Map Reduce engine or Hadoop engine need data on HDFS format to process, We have favorable tools available in market to do this operation.

As an example, Scoop is a tool which converts RDBMS to HDFS. Likewise we have SAP BOD to convert sap system data to HDFS.

Comments

  1. Amazing article! I was confused about Hadoop Training in Chennai, but now got a clear view of the definition. Appreciate your hard work!

    ReplyDelete
  2. I really enjoyed while reading your article, the information you have delivered in this post was damn good. Keep sharing your post with informative content.
    Regards,
    Hadoop Training in Chennai|Hadoop Training Chennai

    ReplyDelete
  3. Thanks for sharing this ,very useful topic

    Hadoop course in Chennai

    ReplyDelete
  4. awesome post presented by you..your writing style is fabulous and keep update with your blogs Big data hadoop online Training Hyderabad

    ReplyDelete
  5. Big Data has made revolution in Data Management and your points prove that, thanks for making my valuable time worth.
    Hadoop Training in Velachery
    Big Data Training in Velachery

    ReplyDelete
  6. It very informative blog and useful article thank you for sharing with us , keep postingBig data hadoop online Course Bangalore

    ReplyDelete
  7. THE INFORMATION YOU PROVIDED IS MUCH APPRECIATED
    online training visit:

    ReplyDelete
  8. I get a lot of great information from this blog. Thanks for sharing this valuable information to our vision.

    Hadoop Training in Chennai | Bigdata Training in Chennai

    ReplyDelete
  9. This comment has been removed by the author.

    ReplyDelete
  10. Thanks for sharing your knowledge with us .This will absolutely going to help me in my future .

    Hadoop Training in Chennai

    Big Data Training in Chennai

    ReplyDelete
  11. Nice Post. Thank you for sharing this in your blog. It is very informative.

    Spark Training | Spark Training Academy

    ReplyDelete
  12. Your blog is very interesting. Have many things to learn in this blog about big data if you want to learn more about big data then we provide both online and offline trainings. Big data training

    ReplyDelete
  13. I actually found this blog and that is amazing thing I enjoy reading this easy to understand stuff. Keep it up. Hadoop Training Chennai | Hadoop Training in Chennai | Big Data Training in Chennai

    ReplyDelete
  14. This is an awesome post. Really very informative and creative contents. These concept is a good way to enhance the knowledge. Thank you for this brief explanation and very nice information.
    Big Data Certification in Chennai | Best Hadoop Training in Chennai | Best hadoop training institute in chennai | Hadoop Course in Chennai | Best Big Data Training in Chennai

    ReplyDelete
  15. Your blog is very interesting. Have many things to learn in this blog about big data if you want to learn more about big data then we provide both online and offline trainings.
    Big Data Hadoop Training in Chennai
    Big Data Hadoop Training in Chennai
    Big Data Hadoop Training in Chennai

    ReplyDelete
  16. It's interesting that many of the bloggers to helped clarify a few things for me as well as giving.Most of ideas can be nice content.The people to give them a good shake to get your point and across the command.
    aws training in chennai

    hadoop training in chennai

    ReplyDelete
  17. Thanks for your sharing such a useful information. this was really helpful to me

    Guest posting sites
    Education

    ReplyDelete
  18. Thanks for sharing this valuable information. Check on the below link if you are looking for best hadoop training in chennai.

    Hadoop Training In Chennai

    ReplyDelete
  19. Very very clear explanation and amazing presentation for hadoop and big data science.
    honor mobile service centre
    honor service center near me
    honor service

    ReplyDelete
  20. A debt of gratitude is in order for your post which accumulates more information about the subject. I read your blog everything is useful.

    ReplyDelete
  21. I have gone through your post and I found it very helpfull. Looking forward to see more post from you.

    Vmware Cloud Migration services

    Database Migration services

    I have gone through your post and I found it very helpfull. Looking forward to see more post from you.

    ReplyDelete

  22. I think about it is most required for making more on this get engaged


    big data course malaysia

    ReplyDelete
  23. Thank you very much for sharing this blog. This blog is very essential and very informative ...
    RPA training in KalyanNagar
    Hadoop training in Bangalore
    Hadoop training in KalyanNagar|
    RPA training in Bellandur

    ReplyDelete
  24. Nice information. Thanks for sharing content and such nice information for me. I hope you will share some more content about. Please keep sharing!
    big data training in chennai

    ReplyDelete
  25. Thanks for the well-written post and I will follow your updates regularly and this is really helpful. Keep posting more like this.
    salesforce Training in Bangalore
    uipath Training in Bangalore
    blueprism Training in Bangalore

    ReplyDelete
  26. Thanks for sharing this valuable information. Check on the below link if you are looking for Best Hadoop Training in Pune
    Big Data Institutes in Pune

    ReplyDelete
  27. Good post!Thank you so much for sharing this pretty post,it was so good to read and useful to improve my knowledge as updated one,keep blogging.
    Big Data Hadoop training in Electronic City

    ReplyDelete
  28. A very nice guide. I will definitely follow these tips. Thank you for sharing such detailed article. I am learning a lot from you.
    salesforce Training in Bangalore
    uipath Training in Bangalore
    blueprism Training in Bangalore

    ReplyDelete
  29. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!.

    aws Training in Bangalore
    python Training in Bangalore
    hadoop Training in Bangalore
    angular js Training in Bangalore
    bigdata analytics Training in Bangalore

    ReplyDelete
  30. This comment has been removed by the author.

    ReplyDelete
  31. Thank you so much for this nice information. Hope so many people will get aware of this and useful as well. And please keep update like this. Data Science Course in Pune

    ReplyDelete
  32. I must appreciate you for providing such a valuable content for us. This is one amazing piece of article. Helped a lot in increasing my knowledge.aws training in bangalore

    ReplyDelete
  33. This is amazing, very rare to find these type of blogs. Must say very well written.

    Hadoop Training

    ReplyDelete

  34. Hi, you know this article is helping for me and everyone and thanks for sharing information Big Data Training Institute in Delhi

    ReplyDelete
  35. Good Post. I like your blog. Thanks for Sharing.
    Hadoop Training in Noida

    ReplyDelete
  36. Nice and good article. It is very useful for me to learn and understand easily.
    Data Science Training in Delhi
    Data Science Training institute in Delhi

    ReplyDelete
  37. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    Big Data, Hadoop Online Training in Pune, Mumbai, Delhi NCR

    ReplyDelete
  38. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    Big Data, Hadoop Online Training in Pune, Mumbai, Delhi NCR

    ReplyDelete
  39. I am really happy with your blog because your article is very unique and powerful for new.
    Hadoop Training in Pune

    ReplyDelete
  40. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    Best Data Science Training Institute in Pune

    ReplyDelete

  41. Wow Very Nice Post I really like This Post. Please share more post. Apache Spark Training Bangalore Boost your career growth by learning the most demanding technology Best Apache Spark Training Institute in Bangalore - Prwatech India's Leading E-Learning Platform offering Advanced Course with 100% Placement Assistance. We also provide Blog, LMS, Regular Course, Self Paced Course, Webinar Session: Marketing, Project: Sell, Technical Support, Lab Service: Sell.

    ReplyDelete
  42. This is most informative and also this post most user friendly and super navigation to all posts.
    Hadoop Training in Pune

    ReplyDelete
  43. Thanks for the post. It was very interesting and meaningful. I really appreciate it! Keep updating stuff like this.
    Big Data Analytics Training Pune

    ReplyDelete
  44. Wow Very Nice Post I really like This Post. Please share more post.
    Apache Spark Training in Bangalore

    ReplyDelete
  45. Really awesome blog!!! I finally found a great post here.I really enjoyed reading this article. Thanks for sharing valuable information.
    Register for a free Demo Sessions

    RPA Ui Path Online Training
    Best Python Online Training
    Online AWS Training
    Online Data Science Training
    Hadoop Online Training

    ReplyDelete
  46. Thank you for sharing the post it is very informative


    Best Hadoop Online Training

    ReplyDelete
  47. Thanks for sharing such a nice Post. I must suggest your readers to Visit Big data course in coimbatore

    ReplyDelete

  48. Awesome blog. I enjoyed reading your articles. This is truly a great read for me.

    Data Scientist Course in pune

    ReplyDelete
  49. I think it could be more general if you get a football sports activity. ExcelR Data Science Courses

    ReplyDelete
  50. Thanks for the detailed blog.The blog conssit of informational content about the topic.I really appreciate your effort in posting such and topic.You may also visit to the GLobal Tech Council to get the best deal.

    Visit- big data certification course

    ReplyDelete

  51. Thanks for the detailed blog.The blog consist of informational data about the topic.I really appreciate the blog posting.You may also visit to the global tech council to get the best deal.

    For more information just click-
    Big Data Certification

    ReplyDelete
  52. Thank you for sharing them ! I hope you will continue to have similar posts to share with everyone !
    We are providing training course with certification. It’s helpful to implement your post as more fair. Our Site informations are given below.
    Devops training in Bangalore

    ReplyDelete
  53. Intelligence is the computational part of the ability to achieve goals in the world. Varying kinds and degrees of intelligence occur in people, many animals and some machines.
    AI Training in Bangalore

    AI Course in Bangalore

    ReplyDelete
  54. This comment has been removed by the author.

    ReplyDelete
  55. Positioning Belts Market 2022 Industry Trends, Sales, Supply, Demand, Analysis & Forecast to 2028
    Summary

    A New Market Study, Titled “Positioning Belts Market Upcoming Trends, Growth Drivers and Challenges” has been featured on fusionmarketresearch.

    This report provides in-depth study of ‘Positioning Belts Market ‘using SWOT analysis i.e. strength, weakness, opportunity and threat to Organization. The Positioning Belts Market report also provides an in-depth survey of major market players which is based on the various objectives of an organization such as profiling, product outline, production quantity, raw material required, and production. The financial health of the organization.

    Positioning Belts Market

    ReplyDelete

Post a Comment

Popular posts from this blog

Zip/Unzip multiple files and also include password for zipped file using SSIS

We have many scenario that we need to Zip many files which we come across and then so some operations like either sending it as a email or just moving zipped file to some other destinations etc. But we were using manual method to zip multiple files. In this post, I tried to create a package which will zip multiple files using SSIS. Here for Zipping files purpose, I'm using 7-ZIP which is free software available in google sites. Download files and install onto your system. First let me show how to Zip on file and later I will show how to zip multiple files using SSIS and 7Zip tool. Compressing Single file. Here I'm trying to Zip one single flat file which is of 40MB size. I kept this file in C:\Documents and Settings\\Desktop\test\source folder. Now to compress this file, I will open my SSIS and I'm dragging and dropping EXECUTE PROCESS TASK from Control Flow. Now right click on Execute Process task and go for edit and select Process option. In process tab,

SSIS: The Value Was Too Large To Fit In The Output Column

I had a SSIS package where I was calling a stored procedure in OLEDB Source and it was returning a “The Value Was Too Large to Fit in the Output Column” error. Well, My Datatype in OLEDB source was matching with my OLEDB Destination table. However, when I googled, we got solutions like to increase the output of OLEDB Source using Advanced Editor option . I was not at all comfortable with their solution as my source, destination and my intermediate transformation all are having same length and data type and I don’t want to change. Then I found that I was missing SET NOCOUNT ON option was missing in Stored Procedure. Once I added it, my data flow task ran successfully. 

How to move multiple files in ssis and also rename simultaneously

There are two ways to achieve this. 1) We can move the flat files and then rename it. 2) While moving files itself, automatic rename should be done. We will do the second type. The criteria is to rename the files while moving from source to destination. So for that, we need FILE SYSTEM TASK to be included. Secondly since we need to move many files, we will use FOR EACH LOOP CONTAINER. To fetch all the files, we can use FOR EACH LOOP task in SSIS. In collection tab, we can select FOREACH FILE enumerator option for fetching files and we can change enumerator configuration Folder option: Points to source where we need to fetch files. Files: will give us idea whether we need to fetch all the files (*.*) or if we give extension like *.txt, it is going to fetch only  .txt files . Once I give Source name in FOR EACH LOOP container, It is going to fetch all the files corresponding to that path. Retrieve file name: This option is used to let the variables mentioned in VARIA